# Maxun Documentation (full text)

> Maxun is the open-source web data platform for gathering, monitoring, and analyzing web data. Turn websites into APIs.

Every page of https://docs.maxun.dev as Markdown, for LLMs and AI agents. For a shorter map of the docs, see https://docs.maxun.dev/llms.txt. Any page is also available as Markdown by adding .md to its URL (for example https://docs.maxun.dev/quickstart.md).

<page url="https://docs.maxun.dev/" title="Maxun Docs: Open-Source Web Scraping & Extraction">

# Maxun Documentation
<h3>Turn any website into live, clean structured data with Maxun</h3>

Maxun is the open-source web data platform for turning the web into structured, reliable data.  
It supports extraction, crawling, scraping, monitoring and search - designed to scale from simple use cases to complex, automated workflows.

## Start Here

- **New to Maxun?** Follow the [Quickstart](/quickstart) to create your first robot in minutes.
- **Not sure which robot to use?** See [Robots Overview](/robot/robots) to compare Extract, Scrape, Crawl, Search and Document robots.
- **Building in code?** Install the [Node.js or Python SDK](/sdks), or use the [REST API](/api/api).
- **Using AI agents?** See [Build with AI](/ai) to [add Maxun to Claude, Cursor, Codex and other agents](/ai/add-to-agent) with MCP or skills, or give them the docs as [llms.txt](https://docs.maxun.dev/llms.txt).
- **Self-hosting?** Install the open-source edition with [Docker Compose](/installation/docker).

## What Maxun Enables
<table>
  <tr>
    <td><strong>Scrape</strong></td>
    <td>Convert full webpages into clean Markdown, HTML and capture screenshots. <a href="/robot/scrape">Learn More</a>.</td>
  </tr>
  <tr>
    <td><strong>Crawl</strong></td>
    <td>Automatically discover and collect data across entire websites with intelligent link following and scoped control. <a href="/robot/crawl">Learn More</a>.</td>
  </tr>
  <tr>
    <td><strong>Agent</strong></td>
    <td>Extract data from websites with AI that autonomously discovers where the data lives. <a href="/robot/agent">Learn More</a>.</td>
  </tr>
  <tr>
    <td width="180"><strong>Extract</strong></td>
    <td>Extract structured data from websites by recording browser actions. <a href="/category/extract">Learn More</a>.</td>
  </tr>
  <tr>
    <td><strong>Search</strong></td>
    <td>Run programmatic web searches and extract results as metadata or full content, with time-based filtering. <a href="/robot/search/search-introduction">Learn More</a>.</td>
  </tr>
  <tr>
    <td><strong>Monitor</strong></td>
    <td>Track websites over time, detect changes, and get notified when they occur. <a href="/monitoring">Learn More</a>.</td>
  </tr>
  <tr>
    <td><strong>Document Extract &amp; Parse</strong></td>
    <td>Extract and parse structured data from documents (PDF, CSV, XLSX, DOCX, JPG and PNG). <a href="/robot/document">Learn More</a>.</td>
  </tr>
</table>

## How To Use Maxun

Maxun is no-code by default, with APIs, CLI, MCP, and SDKs for deeper integrations without changing how extractions are defined.

<table>
  <tr>
    <td width="180"><strong>No-Code Dashboard</strong></td>
    <td>Build, run, and manage robots through Maxun's visual interface. No coding required.</td>
  </tr>
  <tr>
    <td><strong>API</strong></td>
    <td>Integrate Maxun's scraping and extraction capabilities into your applications. <a href="https://docs.maxun.dev/api/api"> Learn More</a>.</td>
  </tr>
  <tr>
    <td><strong>SDK</strong></td>
    <td>Use Maxun programmatically for scraping, extraction, automation and more.<a href="https://docs.maxun.dev/sdks"> Learn more</a>.</td>
  </tr>
  <tr>
    <td><strong>CLI</strong></td>
    <td>Create robots, trigger runs, and retrieve data from your terminal.<a href="https://docs.maxun.dev/category/cli"> Learn more</a>.</td>
  </tr>
  <tr>
    <td><strong>MCP</strong></td>
    <td>Connect Maxun to AI agents through the Model Context Protocol.<a href="https://docs.maxun.dev/category/mcp"> Learn more</a>.</td>
  </tr>
</table>

## What Can You Build?

Teams use Maxun for [e-commerce price and stock monitoring](/usecases/ecommerce_automation), [market research](/usecases/market_research), [lead generation](/usecases/lead_generation), [real estate data](/usecases/real_estate), [content aggregation](/usecases/content_aggregation) and [academic research](/usecases/academic_research), as well as RAG pipelines and AI agents that need live web data.

## Sponsors

<table>
  <tr>
    <td><a href="https://www.webshare.io/?referral_code=wwhk524ynls"><img src="https://github.com/user-attachments/assets/c3a60969-a8f8-4d2c-acef-22bee80c2484" width="200" height="110" /></a></td>
    <td><a href="https://www.swiftproxy.net/?ref=maxun"><img src="https://github.com/user-attachments/assets/a1d2309b-d3bd-45e6-8760-ce1f8dcd18b9" width="200" height="110" /></a></td>
    <td><a href="https://proxy-seller.com/?partner=224ED2B5F5D24719"><img src="https://github.com/user-attachments/assets/ac862778-464d-49ae-af7e-69712d9b74a4" width="200" height="110" /></a></td>
    <td><a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=maxun_github_sponsorship"><img src="https://github.com/user-attachments/assets/f3f7ea4a-a965-4ff9-aab6-8d257da3b18c" width="200" height="110" /></a></td>

  </tr>
  <tr>
    <td><a href="https://mangoproxy.com/?utm_source=github&utm_medium=partner&utm_campaign=getmaxun"><img src="https://github.com/user-attachments/assets/326b5b8f-6a2c-4aa2-94c8-6a29ae23f2ac" width="200" height="110" /></a></td>
    <td><a href="https://go.nodemaven.com/maxunbyopoct"><img src="https://github.com/user-attachments/assets/de295d94-61f4-4c97-88f8-cd45cbf3af84" width="200" height="80" /></a></td>
    <td><a href="https://www.testmuai.com/?utm_medium=sponsor&utm_source=maxun"><img src="https://github.com/user-attachments/assets/6c96005b-85df-43e0-9b63-96aaca676c11" width="200" height="110" /></a></td>
  </tr>
</table>


Continue to the next chapter to understand how Maxun works and how these capabilities are powered.

</page>

<page url="https://docs.maxun.dev/quickstart" title="Maxun Quickstart: Cloud, SDK & Open-Source Setup">

# Get Started

Get started with Maxun in minutes. Sign up for Maxun Cloud, install the Node.js or Python SDK, connect your AI agent with MCP, or self-host the open-source edition with Docker.

## Maxun Cloud

- Sign up at <a href="https://app.maxun.dev/register">https://app.maxun.dev/register</a>.
- Set up your data extraction robot. <a href="/robot/robots">Choose your robot type</a>.
- Name your robot and set it to run regularly, like daily.

That’s it! Most users create their first robot in less than a minute.

## Maxun SDKs

Maxun provides official **Node.js and Python SDKs** for creating and running robots programmatically. 

### Installation

<Tabs>
  <TabItem value="node" label="Node.js">

```bash
npm install maxun-sdk
```

  </TabItem>
  <TabItem value="python" label="Python">

```bash
pip install maxun
```

  </TabItem>
</Tabs>

### Requirements

- API Key from <a href="/api/api">Maxun Dashboard</a>

### Environment Variables

```bash
MAXUN_API_KEY=your-api-key

# For LLM Extraction (optional)
ANTHROPIC_API_KEY=your-anthropic-key
OPENAI_API_KEY=your-openai-key
```

### Quick Start

Create and run a robot programmatically using the Maxun SDK.

<Tabs>
  <TabItem value="node" label="Node.js">

```javascript

const maxun = new Maxun();

const robot = await maxun.scrape('Example page', 'https://example.com');
const result = await robot.run();

console.log(result.markdown);
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
from maxun import Maxun

async with Maxun() as maxun:
    robot = await maxun.scrape("Example page", "https://example.com")
    result = await robot.run()
    print(result.markdown)
```

  </TabItem>
</Tabs>

For more detailed usage, see the [Node.js SDK](/sdk/node-sdk/sdk-overview) and [Python SDK](/sdk/python-sdk/sdk-overview) guides.

## Maxun MCP Server

Connect Claude, Cursor, VS Code, Windsurf, Codex or any other MCP client to Maxun. Your agent can then scrape, crawl, search and extract web data, and run your robots, through natural language.

1. Get an API key from the <a href="/api/api">Maxun Dashboard</a> under **Settings → API Key**.
2. Add the Maxun MCP server to your client:

<Tabs groupId="mcp-client">
  <TabItem value="claude-code" label="Claude Code">

```bash
claude mcp add --transport http maxun https://app.maxun.dev/api/mcp --header "x-api-key: your_api_key_here"
```

  </TabItem>
  <TabItem value="json" label="Cursor and other clients">

```json
{
  "mcpServers": {
    "maxun": {
      "url": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here"
      }
    }
  }
}
```

  </TabItem>
</Tabs>

3. Ask your agent something like *"Scrape https://example.com and summarize it"*.

For other clients, see [Add Maxun to your agent](/ai/add-to-agent). For self-hosted Maxun and the full list of tools, see [MCP Setup](/mcp/setup) and [MCP Tools](/mcp/tools).

## Maxun Community Edition
Maxun is open-source and can run on your own infrastructure. Learn how to <a href="/category/self-host">self-host Maxun</a> with Docker Compose or a local setup.

</page>

<page url="https://docs.maxun.dev/robot/robots" title="Maxun Robots: Extract, Scrape, Crawl & Search">

# What are Maxun Robots?

Maxun robots are automated tools that help you collect data from websites without writing any code. Think of them as your personal web assistants that can navigate websites, extract information, and organize data just like you would manually - but faster and more efficiently.

There are seven types of robots, each designed for a different job.

## Which Robot Should I Use?

| I want to… | Use |
| :--- | :--- |
| Describe the data and let AI find and collect it | Agent |
| Get structured rows (prices, listings) | Extract |
| Get one page as Markdown/HTML | Scrape |
| Get every page of a site | Crawl |
| Find pages for a query | Search |
| Track changes | Monitor |
| Pull data from documents | Document |

## 1. Agent
Describe the data you want in plain English, optionally with a starting URL. The agent finds where the data lives, applies the site's search, filters and sorting, and builds a reusable robot that runs without an LLM call on every run. Available on [Maxun Cloud](https://app.maxun.dev).

Learn more <a href="/robot/agent">here</a>. 
## 2. Extract
Extract emulates real user behavior and captures structured data.
- <a href="/robot/extract/robot-actions">Recorder Mode</a> - Record your actions as you browse; Maxun turns them into a reusable extraction robot.
- <a href="/robot/extract/llm-extraction">AI Mode</a> - Describe the data you want from a page in natural language. Available in the open-source (self-hosted) edition.

Learn more <a href="/category/extract">here</a>.

## 3. Scrape
Scrape converts full webpages into clean Markdown, HTML and can capture screenshots. Ideal for AI workflows, agents, and document processing. 

Learn more <a href="/robot/scrape">here</a>.

## 4. Crawl
Crawl entire websites and extract content from every relevant page, with full control over scope and discovery.

Learn more <a href="/robot/crawl">here</a>.

## 5. Search
Run automated web searches to discover or scrape results, with support for time-based filters.

Learn more <a href="/robot/search/search-introduction">here</a>.

## 6. Monitor
Track websites over time and know when something changes. Your existing robots can become monitors.
Learn more <a href="/monitoring">here</a>.

## 7. Document Extract and Parse
Extract and parse structured data from documents. Supports PDF, CSV, XLSX, JPG, PNG and DOCX files.

Learn more <a href="/robot/document">here</a>.


<!-- ## Robot Actions
Depending on the use-case, a robot should be configured to perform any of the following actions.

1. Capture List
2. Capture Text
3. Capture Screenshot

Learn more about each method in the next chapter. -->

</page>

<page url="https://docs.maxun.dev/robot/scrape" title="Scrape Any Website to Markdown, HTML & Screenshots">

# Scrape

Convert any webpage into clean HTML, LLM-ready Markdown, text, links, AI summaries or screenshots. Ask questions about a page with Smart Queries.

## How It Works

1. Enter the URL you want to scrape.  
2. Choose your output format  
3. Optionally add a **Smart Query** prompt (see below).
4. Run the robot.  

## Output Formats

| Format |  What you get |
|---|---|
| `markdown`  | The page as clean Markdown, ready for an LLM |
| `html`  | The page's HTML |
| `text`  | The page's visible text |
| `links`  | Every link on the page |
| `summary` | An AI-written summary of the page |
| `screenshot-visible`  | A screenshot of the visible part of the page |
| `screenshot-fullpage`| A screenshot of the whole page |


## When to Use Scrape
- Fast content extraction  
- Clean HTML or Markdown for an LLM  

If you need logins, interactions, pagination, or element-level data capture, use <a href="/category/extract">Extract</a> instead.


## Smart Queries

Smart Queries let you attach an optional **natural language prompt** to a scrape robot. After the page is scraped, an LLM analyzes the page content and returns an answer to your prompt - without any extra setup.

### How to Add a Smart Query

When creating a scrape robot, enter your instructions in the **Smart Queries** field:

1. Example: "Click the 'Login' button and extract the user profile data."  
2. Example: "Navigate to the pricing page and list all plan names and prices."  
3. Example: "Find the company's latest blog post title and publication date."

The result is returned in the run output as `promptResult`.

### Output

When a Smart Query is configured, the run result includes an additional `promptResult` field alongside the usual `markdown`, `html`, etc.:

```json
{
  "markdown": "...",
  "html": "...",
  "promptResult": "The pricing plans are: Starter ($9/mo), Growth ($29/mo), Pro ($99/mo)."
}
```

## Using with SDK

Scrape is available through the <a href="/sdks">Maxun SDK</a> for programmatic usage and integration into your applications.

<Tabs>
  <TabItem value="node" label="Node.js">

```javascript

const maxun = new Maxun();

const robot = await maxun.scrape('Example page', 'https://example.com');
const result = await robot.run();

console.log(result.markdown);
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
from maxun import Maxun

async with Maxun() as maxun:
    robot = await maxun.scrape("Example page", "https://example.com")
    result = await robot.run()
    print(result.markdown)
```

  </TabItem>
</Tabs>

## Using with CLI

Scrape is available through the <a href="/category/cli">Maxun CLI</a> for quick data gathering from the terminal.

```bash
maxun robots scrape https://example.com -f markdown
```

</page>

<page url="https://docs.maxun.dev/robot/agent" title="Agents: Web Data Extraction From a Prompt on Maxun Cloud">

# Agent

Agent turns a natural-language request into a reusable data extraction robot. Describe what you want, such as *“Find the 5 cheapest men's tank tops, with name, price, and product URL,”* and optionally provide a starting URL. The agent discovers where the data lives, navigates and configures the website, and builds a robot to collect the information you asked for.

**Autonomous setup, reusable extraction**: The agent handles the discovery and configuration needed to create the robot. Once created, the robot can run repeatedly without requiring an LLM call on every run, making ongoing extraction fast, predictable, and cost-efficient.

> **Note:** Agents are available on [Maxun Cloud](https://app.maxun.dev). The self-hosted open-source edition offers [AI Mode](/robot/extract/llm-extraction), which extracts data from a single page you specify. See [Agent vs AI Mode](/robot/agent-vs-ai-mode) for a comparison.

## Create an agent

1. Open Robots → Create Robot → switch to **Agent** tab.
2. Describe what you need: Specify the data, fields, item count, and any filters or ranking criteria. For example:
   - “The 10 newest remote backend jobs with title, company, and salary”
   - “Pricing plans with name, monthly price, and included features”
   - “The 5 best-selling wireless mice under $50 with name, price, and rating”
3. Add a website URL (optional): Provide a starting point, or let the agent discover the relevant website itself.
4. Run the agent: The agent finds and configures the relevant pages, builds the robot, runs it, and displays the results.

You can also create agents through the [Cloud API](/api/cloud-api#agents), [SDK](/sdk/node-sdk/sdk-extract), [MCP](/mcp/tools.md) and [CLI](/cli/cli-robots).

## What the agent does

1. **Discovers the relevant data.** Starting from your URL or searching the web, the agent identifies the pages most likely to contain the information you need. It can follow relevant links, work across multiple sites, and use pagination when necessary.

2. **Understands your intent.** It translates your request into specific fields, item counts, filters, ranking criteria, and questions to answer. For example, “the cheapest options under $50” defines both a price constraint and a ranking.

3. **Prepares the website.** The agent handles loading states, dismisses cookie banners and pop-ups, and uses the site's search, filters, sorting, and view controls to narrow down the results. These steps are saved as part of the robot's workflow.

4. **Builds a reusable robot.** Maxun identifies the requested information, creates the extraction workflow, and validates it against the actual page. If more items are needed, the robot can navigate the site's pagination to collect them.

5. **Validates the results.** Maxun checks the extracted data against your request, including fields, item counts, duplicates, links, price constraints, and sorting where applicable. 

## Every run

Once the robot has been created, it can be run on demand, scheduled, or monitored like any other Maxun robot.

- **Fast, cost-efficient extraction.** The robot reuses its saved workflow instead of asking an LLM to rediscover and rebuild the extraction process on every run.
- **Answers grounded in extracted data.** If you ask a question, such as “Which plans include SSO?”, the answer is based on the data collected during that run and is available alongside the results through the app, API, and MCP.
- **Transparent filtering.** If a website's filters or sorting controls cannot be applied, Maxun can apply the criteria to the extracted data instead, without misrepresenting how the results were obtained.
- **Resilience to website changes.** Maxun Cloud can automatically adapt robots when a site's layout changes.

## Extract details from individual pages

Enable [Deep Extraction](/deep-extraction) to let the agent follow individual items to their detail pages and extract additional information. This is useful when listing pages contain only basic information and the fields you need are available on each item's own page.


## What agents don't do

- **Sign in.** For pages behind a login, share a logged-in session with the managed Chrome extension.
- **Confirm an age or accept optional cookies.** These are reported, not bypassed.
- **Invent data.** Values the page doesn't show are dropped; when something can't be found, the agent says so.

## Writing effective prompts

Name the fields you want, how many items, and any ranking or filter. For more examples, see <a href="/llm-prompts">LLM Extraction Prompts</a>.

</page>

<page url="https://docs.maxun.dev/robot/crawl" title="Website Crawler: Crawl & Scrape Entire Websites">

# Crawl

Crawl automatically discovers and scrapes multiple pages from a website. Instead of manually specifying each URL, crawl intelligently finds pages and extracts content from each page. Discover pages via sitemaps and links, then get every page as Markdown, HTML or text. Control scope, depth and filters.

## How It Works

1. **Enter Starting URL**: Provide the webpage where the crawl should begin.
2. **Configure Settings**: Set up your crawl preferences.
3. **Automatic Discovery**: The robot finds pages using sitemaps and links.
4. **Content Extraction**: Each discovered page is visited and scraped.

## What Gets Extracted

For each page discovered, crawl extracts
- **Page metadata**: Title, language, description, favicon, and all meta tags
- **HTML content**: Full page HTML
- **Text content**: Clean body text with word count
- **Links**: All links found on the page
- **Status information**: HTTP status code and scrape timestamp
- **Screenshots**: Visible viewport or full-page screenshot per crawled page
- **Summary**: A concise plain-text summary of the page generated by an LLM

## Basic Configuration

**Robot Name**
- Custom name for your crawl robot

**Starting URL**
- The webpage where the crawl begins
- Must be a valid, accessible URL
- All discovered pages will be relative to this starting point

**Max Pages to Crawl**
- Maximum number of pages to crawl
- Pages are discovered and crawled in order of relevance
- Recommended: Start with 10-20 pages to test your configuration

## Advanced Options

### Crawl Scope

Choose how broadly the robot should crawl from your starting URL:

**Domain Mode**
- Crawls only pages on the exact same domain
- Example: Starting at `blog.example.com` stays on `blog.example.com`
- Best for: Focused crawls on a specific subdomain

**Subdomain Mode**
- Crawls the domain and all its subdomains
- Example: Starting at `example.com` includes `blog.example.com`, `shop.example.com`, etc.
- Best for: Comprehensive site crawls across multiple sections

**Path Mode**
- Crawls only pages under the same path as the starting URL
- Example: Starting at `example.com/blog/` stays within `/blog/` path
- Best for: Crawling specific sections like documentation or blog categories

### Max Depth

**Maximum Crawl Depth**
- Controls how many "levels" deep from the starting URL the crawler should go
- Each click or navigation from one page to another counts as one level
- Higher depth values discover more pages but increase crawl time

### URL Filtering
**Note:** URL Filtering is currently in development and not fully enforced.

**Include Paths**
- Regex patterns for URLs to include in your crawl
- Only URLs matching these patterns will be crawled
- Leave empty to include all URLs within your scope
- Example: `/blog/[0-9]{4}/.*` for dated blog posts

**Exclude Paths**
- Regex patterns for URLs to exclude from your crawl
- URLs matching these patterns will be skipped
- Example: `.*/admin/.*` to skip admin pages
- Useful for avoiding login pages, tag pages, or duplicate content

**Pattern Tips:**
- Patterns use JavaScript regular expressions (regex)
- The `.*` matches any characters (wildcard)
- Use `\\.` to match literal dots in URLs
- Patterns are case-sensitive by default

### Discovery Options

**Use Sitemap**
- When enabled, fetches and parses the website's sitemap.xml
- Automatically follows nested sitemaps
- Recommended for sites with well-maintained sitemaps

**Follow Links**
- When enabled, extracts all links from each visited page
- Crawls pages that match your scope and filters
- Recommended when sitemap is incomplete or unavailable

**Best Practice**: Enable both options for comprehensive discovery. The robot will combine URLs from both sources and remove duplicates.

### Robots.txt Compliance
When enabled, the crawler respects robots.txt directives. Recommended for ethical crawling of third-party websites. By default enabled.

## Example Configurations

**Simple Blog Crawl**
```javascript
{
  mode: 'path',
  limit: 50,
  useSitemap: true,
  followLinks: true
}
```

**Filtered Blog Crawl (Advanced)**
```javascript
{
  mode: 'path',
  limit: 50,
  includePaths: ['/blog/[0-9]{4}/.*'],
  excludePaths: ['.*/tag/.*', '.*/author/.*'],
  useSitemap: true,
  followLinks: true
}
```

**Full Site Crawl**
```javascript
{
  mode: 'subdomain',
  limit: 100,
  excludePaths: ['.*/admin/.*', '.*/login.*'],
  useSitemap: true,
  followLinks: true
}
```

**Documentation Crawl**
```javascript
{
  mode: 'path',
  limit: 200,
  useSitemap: true,
  followLinks: false
}
```

##  When to Use Crawl 

1. You need to scrape multiple pages from a website
2. You want to discover pages automatically without listing URLs manually
3. You're extracting similar content across many pages (blog posts, product pages, documentation)
4. The website has a clear structure or sitemap

## When Not to Use Crawl

1. You only need data from a single page (use Extract or Scrape instead)
2. You need complex interactions like logins or form submissions
3. You need to extract structured data in a specific format (use Extract)
4. You need to control the exact order pages are visited

For complex workflows with user interactions, use Extract instead.

## Using with SDK

Crawl is available through the <a href="/sdks">Maxun SDK</a> for programmatic usage and integration into your applications.

<Tabs>
  <TabItem value="node" label="Node.js">

```javascript

const maxun = new Maxun();

const robot = await maxun.crawl('Example docs', 'https://docs.example.com', { limit: 20 });

const result = await robot.run();

for (const page of result.crawlData) {
  console.log(page.metadata.url);
  console.log(page.markdown?.slice(0, 200));
}
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
from maxun import Maxun

async with Maxun() as maxun:
    robot = await maxun.crawl("Example docs", "https://docs.example.com", limit=20)
    result = await robot.run()
    for page in result.crawl_data:
        print(page["metadata"]["url"])
        print(page.get("markdown", "")[:200])
```

  </TabItem>
</Tabs>

## Using with CLI

Crawl is available through the <a href="/category/cli">Maxun CLI</a> for quick data gathering from the terminal.

```bash
maxun robots crawl https://docs.example.com --limit 20 --include "/docs/*" -n "Docs Crawler"
```

</page>

<page url="https://docs.maxun.dev/robot/agent-vs-ai-mode" title="Agent vs AI Mode: Building Robots From a Prompt">

# Agent vs AI Mode

Maxun offers two ways to use natural language to extract data from websites.

- **[Agent](/robot/agent)** on Maxun Cloud: describe the data you want, optionally provide a URL, and let the agent discover where the data lives and how to extract it.
- **[AI Mode](/robot/extract/llm-extraction)** in the self-hosted open-source edition: provide a URL and describe the data you want to extract from that specific page.

Both create reusable robots that can be run repeatedly without making LLM calls on every run. You can schedule, monitor, and use them through the API, SDK, MCP and CLI.

## Two different approaches: autonomous discovery vs. page-specific extraction

**Agent is designed for autonomous discovery.** You describe the outcome you want, not necessarily where to find it. The agent can search the web, identify relevant pages, follow links, apply a website's search, filters and sorting, and determine how to collect the requested data. It handles the discovery and setup, then saves a reusable robot for subsequent runs.

**AI Mode is designed for focused extraction.** You provide the exact page containing your data and describe what you need in natural language. Maxun builds a robot to extract the requested information from that page. It doesn't independently search the web or navigate to other pages to find missing data.

The distinction is simple: **Agent figures out where and how to get the data; AI Mode extracts the data from the page you specify.**

## At a glance

| Capability | Agent | AI Mode |
|---|---|---|
| **Available on** | Maxun Cloud | Self-hosted open source |
| **Website URL** | Optional. Searches the web when no URL is provided | Required. You specify the page to extract from |
| **Page discovery** | Discovers relevant pages and can follow links to locate the data | No autonomous discovery; reads only the specified page |
| **Multiple websites** | Can collect data across multiple sites in one prompt | Limited to the specified page |
| **Pagination** | Follows pagination to collect the requested items | Can scroll or use the page's own Next / Load more controls when you request more items than initially available |
| **Site search, filters and sorting** | Can apply the website's own controls and replay them on subsequent runs | Not applied automatically. Provide a URL that already displays the view you need |
| **Detail pages** | Supports [Deep Extraction](/deep-extraction) to follow individual items to their detail pages | Does not navigate to detail pages |
| **Questions in the prompt** | Can answer questions using the data collected during the run, such as “Which plans include SSO?” | Extracts the requested data; doesn't answer additional questions about it |
| **Result validation** | Checks results against the request, including fields, item counts, duplicates, links and applicable constraints | Checks the extraction against the specified page; unavailable fields are reported |
| **When data is missing** | Can search for other relevant pages and reports anything it couldn't find | Reports missing data or stops with an error; doesn't look elsewhere |
| **Website layout changes** | Maxun Cloud can automatically adapt the robot | Not supported automatically. Create a new AI Mode robot |
| **LLM configuration** | Managed by Maxun | Configure your own supported provider, including Anthropic, OpenAI, OpenAI-compatible servers, or Ollama |

## Which one should you use?

**Use Agent** when you know what data you want but don't know exactly where it lives, or when finding the right page and applying website controls are part of the task.

Examples:
- “Find the 5 cheapest 4K monitors on this store.”
- “Find Acme's pricing plans and identify which ones include SSO.”
- “Collect the latest remote backend jobs from these websites.”

**Use AI Mode** when you know the page containing the data and want to describe the extraction in natural language.

Examples:
- “Extract the product names, prices and ratings from this category page.”
- “Extract the pricing tiers and included features from this pricing page.”
- “Extract the names and contact details listed in this directory.”

For workflows that require logging in, filling forms or clicking through multiple steps before the data appears, use [Recorder Mode](/robot/extract/robot-actions).

## Example

Prompt: *“Extract 15 products with their name, price and ID.”*

- **Agent:** Without a URL, it searches for the relevant store, locates a product listing, and collects 15 products, following pagination when needed. It then checks the results against the request.
- **AI Mode:** Given `https://example.com/products`, it extracts products from that page. If only 12 are initially visible, it can use the page's pagination or load-more controls to retrieve the remaining items. If the requested data isn't available, it reports the problem rather than searching elsewhere.

See also: [Cloud vs Open Source](/cloud-vs-oss).

</page>

<page url="https://docs.maxun.dev/robot/document" title="Document Robots: Extract & Parse PDFs and Files">

# Document Robots

Maxun can work with documents in two ways:
1. **Structured Extraction**: Extract specific data fields from documents.
2. **Document Conversion**: Convert entire documents into a clean, structured format.

Note: Document robots are in beta.

## Supported Files

Document robots accept **PDF**, **CSV**, **XLSX**, **JPG**, **PNG**, and **DOCX** files, up to **10 MB**. For multi-sheet XLSX files, each sheet is treated as a separate page.

## Extract

Upload a PDF, CSV, XLSX, JPG, PNG or DOCX file and tell Maxun what information you want from it. Maxun figures out the structure automatically and saves it as a reusable robot — so you can run it on new documents any time.


### How Document Extraction Works

1. Upload a sample document (PDF, CSV, XLSX, JPG, PNG, or DOCX).
2. Describe what you want to extract (e.g. *"invoice number, vendor name, and total amount"*).
3. Maxun creates a robot that can extract those fields from any similar document.
4. Run the robot whenever you have a new file to process.

### When to Use Extract

- You need specific fields from documents like invoices, contracts, or offer letters.
- You receive the same type of document repeatedly and want to automate processing it.

---

## Parse

Upload a PDF, CSV, XLSX, JPG, PNG, or DOCX file and convert it into clean Markdown, HTML, a list of links, or an AI-generated summary. No configuration needed — just pick your format and run.


### How Document Parsing Works

1. Upload a document (PDF, CSV, XLSX, JPG, PNG, or DOCX).
2. Choose your output format - Markdown, HTML, Links, or Summary.
3. Maxun converts the document and makes it available for every run.

### Output Formats

- **Markdown** - The full document as clean, readable Markdown. Great for feeding into AI tools.
- **HTML** - The document as structured HTML.
- **Links** - Every URL found in the document, deduplicated.
- **Summary** - A concise, AI-generated summary of the document. Requires an LLM provider (Ollama, Anthropic, or an OpenAI-compatible provider), configured when you create the robot.

> Note: Self-hosted installations use your configured LLM. Ollama runs locally without an API key; hosted providers require an API key when creating the robot or via `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` in your server environment. Maxun Cloud automatically selects the best AI for the task.

### When to Use Parse

- You want to feed a document into an AI tool or pipeline.
- You need the content of a document in a readable, structured format.


## Using with SDK

Document Robots are available through the <a href="/sdks">Maxun SDK</a> for programmatic usage and integration into your applications.

```javascript
// Extract specific fields
const invoiceRobot = await maxun.documents.extract('Invoice Extractor', './invoice.pdf', 'Extract invoice number, vendor name, and total amount');

// Convert a document to Markdown
const parseRobot = await maxun.documents.parse('Report Parser', './report.pdf');
```

See [Node.js Document](/sdk/node-sdk/sdk-document) and [Python Document](/sdk/python-sdk/sdk-document) for all options.

## Using with CLI

Document Robots are available through the <a href="/category/cli">Maxun CLI</a> for quick data gathering from the terminal.

```bash
# Extract specific fields from a PDF, CSV, or XLSX file
maxun robots doc-extract invoice.pdf \
  --prompt "Extract invoice number, vendor name, and total amount" \
  --name "Invoice Extractor"

# Convert a document to Markdown, HTML, links, and a summary
maxun robots doc-parse report.pdf \
  --formats "markdown,html,links,summary" \
  --name "Report Parser"

# Parse a spreadsheet
maxun robots doc-parse data.xlsx \
  --formats "markdown" \
  --name "Spreadsheet Parser"
```

</page>

<page url="https://docs.maxun.dev/robot/extract/robot-actions" title="Recorder Mode: Point-and-Click Web Data Extraction">

# Recorder Mode

Recorder Mode lets you record your actions into a workflow. Show the Recorder what you want to capture, and a robot will watch and learn.

Recorder Mode is the most precise way to build an Extract robot. Use it when you need exact fields, multi-step flows, logins, form filling or pagination. Robots built in Recorder Mode can also be [duplicated](/robot/extract/robot-duplicate) for similar pages and [retrained](/robot/extract/robot-retrain) when a site changes.

Depending on the use case, a robot should be configured to perform any of the following actions.

## 1. Capture List
Capture List should be used to capture bulk data. Example: Extract products from <a href="https://producthunt.com">producthunt.com</a>. Capture List involves four steps:
1. Select the product/item to capture.
2. Select fields inside the selected product/item.
3. Show the robot how to handle pagination.
4. Set a limit, i.e. the number of items to capture.

Supported pagination types:
- **Click "Next"** to move to the next page
- **Click "Load more"** to load more items on the same page
- **Scroll down** for infinite scrolling pages
- **Scroll up** for pages that load items upwards
- **No more items**, when everything is on one page

Check out this video to understand how to create a robot with capture list

<iframe width="560" height="315" src="https://www.youtube.com/embed/ZXGQEwQN7yI?si=PaNzVTbWn9z4Vh0E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>


## 2. Capture Text
Capture Text should be used to extract specific text content. Useful to get individual data and when you do not want to bulk scrape.

Check out this video to understand how to create a robot with capture text
<iframe width="560" height="315" src="https://www.youtube.com/embed/ZXGQEwQN7yI?si=k-etTEyhx_a9yFOr&amp;start=275" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## 3. Capture Screenshot
Capture Screenshot should be used to extract screenshots of websites. Currently supported screenshots include:
1. Full page screenshots
2. Visible section screenshots

Check out this video to understand how to create a robot with capture screenshot
<iframe width="560" height="315" src="https://www.youtube.com/embed/ZXGQEwQN7yI?si=Lqlu94nDl1CWBwPc&amp;start=195" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## What Recorder Mode Supports

- Clicking buttons and links
- Filling out forms, dropdowns, radio buttons, checkboxes, dates and times
- Logging in to websites (see [Extract Behind Login](/extract-login))
- Extracting data inside iFrames and Shadow DOM
- Capturing image and file URLs

## Using with SDK

The same capture actions are available in code through the <a href="/sdks">Maxun SDK</a>, using CSS selectors:

```javascript
const robot = await maxun.extract('Products', 'https://shop.example.com')
  .captureList({ selector: 'article.product_pod', maxItems: 20 })
  .build();
```

See [Node.js Extract](/sdk/node-sdk/sdk-extract) and [Python Extract](/sdk/python-sdk/sdk-extract) for the full builder API.

</page>

<page url="https://docs.maxun.dev/robot/extract/llm-extraction" title="AI Mode: Extract Web Data With Natural Language">

# AI Mode

AI Mode turns a web page and a simple description of the data you need into a reusable data extraction robot. Just enter a URL and a prompt, such as *“Extract the five men's basketball shoes with their names, prices, ratings, and product URLs.”* Maxun figures out how to extract the requested data and builds a robot that can run repeatedly.

**LLM calls are only needed when creating the robot, not on every run.** Once created, the robot can extract data from the page efficiently without recurring AI costs. You can schedule runs, monitor changes, and use your robots through the API, SDK, and CLI.

AI Mode is part of the self-hosted open-source edition and uses the LLM you configure. On Maxun Cloud, robots built from a prompt are [Agents](/robot/agent). See [Agent vs AI Mode](/robot/agent-vs-ai-mode) for how they differ.

## Create a robot with AI Mode

1. Open **Robots → Create Robot → Extract** and switch to **AI Mode**.
2. **Website URL** (required): the page that holds the data. AI Mode reads this page only.
3. **Prompt**: describe the data, the fields you want and, if it matters, how many items. For example:
   - "Extract the first 20 product names and prices"
   - "Get company names, descriptions and batch"
   - "Extract article headlines, authors and publication dates"
4. **Model settings**: choose the LLM provider and model. See [LLM providers](#llm-providers).
5. Click **Create and run**. Maxun builds the robot, runs it once and opens the results.

## How it works

AI Mode turns a URL and a natural-language description of the data you need into a reusable data extraction robot.

1. **Specify the page and the data.** Enter the URL of the page containing your data and describe what you want to extract, including specific fields or a desired number of items.

2. **Understand your request.** Maxun interprets your instructions to identify the information you need, rather than simply extracting the largest list or collection on the page.

3. **Extract the data.** Maxun identifies the requested information on the specified page and builds a robot to extract it. It can reveal content hidden behind cookie banners, modals, accordions, or lazy loading. If you request more items than are initially visible, it can use the page's own pagination or load-more controls to retrieve them.

4. **Validate and save the robot.** Maxun checks the extraction and saves a reusable robot that can run against your specified URL. If some requested fields aren't available, the robot records what's missing. If the requested data cannot be found, creation stops with an explanation.

5. **Run it whenever you need.** Once created, the robot can be run repeatedly without making LLM calls on every run. Schedule it, monitor extraction results, or integrate it through the API, SDK, and CLI.

### Designed for reliable extraction, not autonomous browsing

AI Mode is built to create focused data extraction robots, not autonomous agents that explore the web to complete open-ended tasks.

- **Page-specific:** Extracts data from the URL you provide, rather than searching other websites for missing information.
- **Predictable:** Stays within the specified page and its permitted interactions. It doesn't independently browse elsewhere or follow unrelated links.
- **Intent-driven:** Extracts the fields and items you describe, rather than guessing what data might be useful.
- **Reusable and cost-efficient:** AI helps build the robot; subsequent runs reuse it without recurring LLM calls for extraction.

If the information isn't available on the specified page, Maxun tells you instead of silently searching elsewhere or inventing missing values.


## LLM providers

Choose a provider under **Model settings** when you create the robot, or pass it from the SDK, API or CLI.

| Provider | API key | Default model | Notes |
| :-- | :-- | :-- | :-- |
| **Anthropic** | Required: in Model settings or `ANTHROPIC_API_KEY` | `claude-sonnet-5` | Get a key from the <a href="https://console.anthropic.com/">Anthropic Console</a> |
| **OpenAI / OpenAI-compatible** | Required: in Model settings or `OPENAI_API_KEY` | `gpt-5.6-terra` | Set a base URL to use any OpenAI-compatible server |
| **Ollama** | Not needed locally. Ollama Cloud: `OLLAMA_API_KEY` | `llama3.2` | Pull the model first (`ollama pull llama3.2`), or set another one |

Choosing a model in Model settings overrides the default. Any model that follows instructions and returns JSON reliably works; larger models give better results on complex pages.

### Ollama

- **Local**: Maxun connects to `http://localhost:11434`. If Maxun runs in Docker and Ollama runs on your machine, set `OLLAMA_BASE_URL=http://host.docker.internal:11434`.
- **Ollama Cloud**: set the base URL, your API key, and a model that Ollama Cloud serves:

```bash
OLLAMA_BASE_URL=https://ollama.com
OLLAMA_API_KEY=your-ollama-api-key
OLLAMA_MODEL=gpt-oss:20b
```

If the model doesn't exist on the server, robot creation fails with Ollama's own message (for example `model 'llama3.2' not found`) and what to check.

See <a href="/installation/environment_variables">Environment Variables</a> for all settings.

## When to use AI Mode

- You want data from a page without learning the Recorder.
- The data is a list (products, jobs, articles, listings) or a few values on a page (a price, an address, a date).
- You know the page that shows the data.

## When to use Recorder Mode instead

- You need to log in, fill in forms or click through several steps before the data appears.
- The data is spread across several pages, or needs the site's search, filters or sorting applied first.
- You need exact control over every step of the robot.

## Using with the SDK

```javascript
const robot = await maxun.extract('YC companies', 'https://www.ycombinator.com/companies', {
  prompt: 'Extract the first 20 company names and descriptions',
  llmProvider: 'anthropic',
  llmApiKey: 'your-llm-api-key',
});
const result = await robot.run();
```

The URL is required. See [Node.js Extract](/sdk/node-sdk/sdk-extract) and [Python Extract](/sdk/python-sdk/sdk-extract).

## Using with the CLI

```bash
maxun robots extract \
  -p "Extract all product names and prices" \
  -u "https://example.com/shop" \
  -n "Shop Extractor"
```

See [CLI robots](/cli/cli-robots).

## Writing effective prompts

Name the fields you want and, if it matters, how many items. For more examples, see <a href="/llm-prompts">LLM Extraction Prompts</a>.

</page>

<page url="https://docs.maxun.dev/robot/extract/robot-duplicate" title="Duplicate a Robot to Scrape Similar Pages in Bulk">

# Robot Duplication

Robot duplication is useful to extract data from pages with the <b>same structure without training a new robot</b>.

Robot duplication is only available for <a href="/robot/extract/robot-actions">Extract Recorder Mode</a>.

## When to Duplicate a Robot
1. The new page has the same structure as the existing page.
2. You want to extract the same data as the existing page.

Example: If you've created a robot for <a href="https://www.producthunt.com/topics/chrome-extensions">producthunt.com/topics/chrome-extensions</a>, you can duplicate it to scrape similar pages like <a href="https://www.producthunt.com/topics/sports">producthunt.com/topics/sports</a> without training a robot from scratch.

Using robot duplication, you can bulk extract the same data from thousands of pages of the same website, without writing code.

## When Not to Duplicate a Robot
1. The new page does not have the same structure as the existing page.
2. You don't want to extract the same data as the existing page even if the pages are structurally the same.

Example: If you've created a robot for <a href="https://www.producthunt.com/topics/chrome-extensions">producthunt.com/topics/chrome-extensions</a>, you should not duplicate it to scrape pages like <a href="https://github.com">github.com</a>.
If you do so, you will get no data.

## Duplicate With the API and CLI

You can also duplicate robots programmatically:

### REST API

`POST /api/v1/robots/{id}/duplicate` with a `url` in the body. See [Cloud API](/api/cloud-api#update-duplicate-and-delete).

### CLI

`maxun robots duplicate <robot-id> --url <new-url>`. See [CLI Robots](/cli/cli-robots).

## See Robot Duplication In Action
<iframe width="560" height="315" src="https://www.youtube.com/embed/fdW8VPcAsN8?si=wqynEzmy9IbOsciG" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

</page>

<page url="https://docs.maxun.dev/robot/extract/robot-retrain" title="Retrain a Robot When a Website Changes">

# Retrain a Robot

Websites change their layout over time. When that happens, a robot trained on the old layout can stop returning data or return the wrong fields. Retraining lets you fix the robot in minutes instead of building a new one.

## Why Retrain?

1. Update a robot’s logic with just a point and click.
2. Adapt to website changes in minutes.
3. No need to start from scratch.


## When Do I Need to Retrain?

- The website changed its layout or design and runs return empty or wrong data.
- You want to capture new fields or remove fields you no longer need.
- You want to change pagination or the steps the robot performs before extracting.

> **Tip:** On [Maxun Cloud](/cloud-vs-oss), robots automatically adapt to most website layout changes, so you need to retrain far less often.

## How to Retrain
This feature is exclusively available for <a href="/robot/extract/robot-actions">Extract Robots built using Recorder Mode</a>.

1. Open Robot Options: Navigate to your dashboard and find the robot you wish to update.
2. Select Retrain: Open the options dropdown menu (the three dots ...) and select "Retrain".
3. Update Logic: The Recorder will open. Simply point and click to re-select the data you need or record new interactions.
4. Save & Resume: Once finished, your robot will immediately begin using the updated logic for all future runs.

</page>

<page url="https://docs.maxun.dev/robot/search/search-introduction" title="How Web Search Robots Work in Maxun">

# Search Overview

Search performs web searches and scrapes content from search results. Results come from DuckDuckGo. Instead of manually searching and visiting each result, search robots automate the entire process - from running queries to scraping content.

## How It Works

1. **Enter Search Query**: Describe what you're looking for in natural language.

2. **Configure Search Settings**: Define your search parameters:
   - Number of results to retrieve
   - Search mode (discover or scrape)
   - Time filters (day, week, month, year)

3. **Create & Run**: The robot runs your search, collects results, and optionally scrapes full content from each result page.

## Search Modes

**Discover Mode**
- Returns search result metadata only (title, URL, description)
- Fast and lightweight
- Perfect for research and URL discovery

**Scrape Mode**
- Returns full page content from each search result
- Includes metadata, HTML, text, and links
- Ideal for content analysis and AI workflows

## What Gets Extracted

**Discover Mode Returns**
- Search result title and description
- Result URL

**Scrape Mode Returns All Above Plus**
- Full page metadata (title, language, meta tags, favicon)
- Complete HTML content
- Clean text content with word count
- All links found on the page
- HTTP status code
- Visible viewport or full-page screenshot per result
- A concise plain-text summary per result generated by an LLM

## When to Use Search

1. You need to gather information from multiple sources on a topic
2. You want to scrape content from top search results automatically
3. You need to monitor search rankings for specific queries

## When Not to Use Search

1. You already know the exact URLs you want to scrape (use Crawl or Scrape)
2. You need results from a specific search engine other than DuckDuckGo

For known URLs, use Crawl for multiple pages or Scrape for single pages.

## Using with SDK

Search robots are available through the <a href="/sdks">Maxun SDK</a> for programmatic usage and integration into your applications.

<Tabs>
  <TabItem value="node" label="Node.js">

```javascript
const robot = await maxun.search('AI news', 'AI model releases', { timeRange: 'week' });
const result = await robot.run();
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
robot = await maxun.search("AI news", "AI model releases", time_range="week")
result = await robot.run()
```

  </TabItem>
</Tabs>

See [Node.js Search](/sdk/node-sdk/sdk-search) and [Python Search](/sdk/python-sdk/sdk-search) for all options.

## Using with CLI

```bash
maxun robots search "Latest AI news" --mode discover --limit 10 -n "AI News"
```

See [CLI Robots](/cli/cli-robots) for all flags. To configure query, limits, modes and time filters, see [Search Configuration](/robot/search/search-configuration).

</page>

<page url="https://docs.maxun.dev/robot/search/search-configuration" title="Search Configuration: Modes, Limits & Time Filters">

# Search Configuration

Configure how your search robot queries and processes search results.

<!-- ![Search Configuration Settings](search_config.png) -->

## Search Query

**Query**
- Enter your search terms in natural language
- Same as you would type in a search engine
- Be specific for better results
- Example: "Latest AI Breakthroughs 2025"

## Result Limits

**Limit**
- Maximum number of search results to process
- Results are returned in order of search engine ranking
- Typical range: 5-50 results

> **Note:** The actual number of results may be lower if fewer results are available for your query.

## Search Mode

Choose how deeply you want to process search results:

**Discover Mode**
- Returns only search result metadata
- Use when you need URLs and descriptions

**Scrape Mode**
- Navigates to each result page and extracts full content
- Use when you need complete page content

<!-- ![Mode Comparison](mode_comparison.png) -->

## Time Filters

Filter search results by when they were published or indexed.

**Time Range Options:**
- **Day**: Results from the last 24 hours
- **Week**: Results from the last 7 days
- **Month**: Results from the last 30 days
- **Year**: Results from the last 12 months
- **All time**: No time restriction (default)

Great for finding recent news, fresh content, or timely information.

## Example Configurations

**Quick Research (Discover Mode)**
```javascript
{
  query: "react performance optimization",
  limit: 10,
  mode: 'discover',
  filters: {
    timeRange: 'year'
  }
}
```

**Deep Content Analysis (Scrape Mode)**
```javascript
{
  query: "machine learning tutorials",
  limit: 20,
  mode: 'scrape',
  filters: {
    timeRange: 'month'
  }
}
```

**Breaking News Monitoring**
```javascript
{
  query: "artificial intelligence regulations",
  limit: 15,
  mode: 'scrape',
  filters: {
    timeRange: 'day'
  }
}
```

**Competitive Research**
```javascript
{
  query: "best crm software for startups",
  limit: 30,
  mode: 'scrape',
  filters: {
    timeRange: 'year'
  }
}
```

## Tips for Better Results

**Write Clear Queries**
- Be specific about what you're looking for
- Include relevant keywords and context
- Use quotes for exact phrases

**Choose the Right Mode**
- Use discover mode for URL collection and quick research
- Use scrape mode when you need full content for AI processing or analysis

**Apply Time Filters**
- Use recent time filters for news and trending topics
- Use longer time ranges for evergreen content and comprehensive research

**Set Appropriate Limits**
- Start with smaller limits (5-10) to test your query
- Increase limits for comprehensive research

</page>

<page url="https://docs.maxun.dev/runs" title="Robot Runs: Manual, Scheduled & API">

# Runs

Runs are the core functionality of Maxun robots. Each run represents a complete cycle where the robot performs its tasks based on the configuration and training provided by the user at the time of robot creation. A successful run contains all the extracted data, fulfilling the primary objective of the robot.

## Execution Options

Robot runs can be initiated in three different ways:

**1. Manual Runs**: Users can run a robot directly from Maxun dashboard.

**2. Scheduled Runs**: Automate runs by scheduling them to execute at specific times or intervals. 

**3. API Runs**: Trigger runs programmatically via API calls, enabling integration with external systems.

## Ways to Run a Robot

| Where | How |
|---|---|
| Dashboard | Click **Run** on any robot |
| Schedule | Set an interval in the robot's [schedule settings](/robot-schedule) |
| REST API | `POST /api/v1/robots/{id}/runs` ([Cloud API](/api/cloud-api#runs)) |
| Node.js SDK | `const result = await robot.run();` ([Robot Management](/sdk/node-sdk/sdk-robot)) |
| Python SDK | `result = await robot.run()` ([Robot Management](/sdk/python-sdk/sdk-robot)) |
| CLI | `maxun run <robot-id>` ([Running Robots](/cli/cli-run)) |
| MCP | Ask your AI client to run a robot with the `maxun_run_robot` tool ([MCP Tools](/mcp/tools)) |

## Viewing and Exporting Results

Every run is saved with its extracted data. From the dashboard you can view a run's output and export it as CSV or JSON. You can also send results automatically to [webhooks](/api/webhooks), [Google Sheets](/integrations/gsheet), [Airtable](/integrations/airtable) or [n8n](/integrations/n8n).

To get alerts when the data changes between runs, turn on [Monitoring](/monitoring).

</page>

<page url="https://docs.maxun.dev/robot-schedule" title="Schedule Robots to Run Automatically">

# Schedule

Robots can be scheduled to run automatically at specific time intervals, so your data stays fresh without manual runs. In the dashboard, scheduling is configured in the robot's settings:

![Maxun Schedule Settings](./schedule_settings.png)

## 1. Run once every

A robot can be scheduled to run once every provided time duration:

- Minutes
- Hours 
- Days
- Weeks 
- Months

## 2. Start from

A robot can be scheduled to start at any day of the week.

## 3. In Between

A robot can be scheduled to run between a specified time interval.

## 4. Timezone

A robot can be scheduled to run across different timezones.

![Maxun Schedule Settings Set](./schedule_set.png)

Once the robot has been scheduled it will run successfully at the specified time. You can delete the schedule at any time.

## Schedule With the SDK

<Tabs>
  <TabItem value="node" label="Node.js">

```javascript
// Every day at 9:00 in Kolkata time
await robot.schedule({ runEvery: 1, runEveryUnit: 'DAYS', atTimeStart: '09:00', timezone: 'Asia/Kolkata' });
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
# Every day at 9:00 in Kolkata time
await robot.schedule(run_every=1, run_every_unit="DAYS", at_time_start="09:00", timezone="Asia/Kolkata")
```

  </TabItem>
</Tabs>

See [Node.js Robot Management](/sdk/node-sdk/sdk-robot) and [Python Robot Management](/sdk/python-sdk/sdk-robot) for all scheduling options.

## Schedule + Monitoring

Combine a schedule with [Monitoring](/monitoring) to get notified when a website changes, for example to track competitor prices daily.

</page>

<page url="https://docs.maxun.dev/monitoring" title="Website Change Monitoring & Alerts">

# Monitoring

Monitoring lets you track a website over time and get notified when it changes. You can monitor an entire website, or just a specific part of it. It works with **Extract**, **Scrape**, and **Crawl** robots, and is available through the no-code dashboard, the SDK, the CLI, and the API.

Every time a monitored robot runs, its output is compared against the previous run, so you can see exactly what's different.

## What can be monitored

What you're monitoring depends on the robot type:

* **Extract** - monitor specific parts of a page: text, lists, or screenshots.
* **Scrape** - monitor the entire page, captured in whichever output formats you've configured, including screenshots.
* **Crawl** - monitor an entire website as it crawls across pages.

Screenshot comparison is a **Cloud-only** feature and is not available in the open-source (OSS) version. Crawl monitoring is also **Cloud-only**. All other content formats are supported on both Cloud and OSS.

## Enabling monitoring

You can turn monitoring on in two places:

1. **At creation time** - when creating a robot, check the monitoring checkbox in the creation flow.
2. **From robot settings** - open an existing robot's settings and enable monitoring there. This is also where you can turn it off.

## Viewing changes

Once monitoring is enabled, changes are shown in the **Runs** tab for each individual run.

When a run contains changes compared to the previous run, a **Changed** button will appear next to that run. Click **Changed** to view exactly what changed between the two runs.

## AI Mode

Instead of just comparing raw output, you can enable AI Mode and describe what you actually care about in plain language. This is useful when you don't want to be notified about every change - only specific ones that matter to you.

For example, instead of monitoring every value on a page, you could prompt:

> "Tell me when births increase to 200 today"

This works across all monitor types (Extract, Scrape, and Crawl) and uses AI to evaluate each run against your prompt, rather than doing a plain diff.

## Scheduling

Monitored robots can be scheduled to run automatically at whatever interval you need, so you don't have to trigger runs manually to catch changes. See [Scheduling](./robot-schedule) for details on setting up recurring runs.

## Alerts

When monitoring detects a change, you can enable email alerts from the **Monitoring** section in your Robot settings. After enabling monitoring, turn on **Email Alerts** and optionally enter the email address you want to receive alerts at.

If the email field is left empty, alerts will be sent to the email address associated with your account.

Email alerts are a **Cloud-only** feature and are not available in OSS.

## Availability

| Interface           | Supported |
| ------------------- | --------- |
| Dashboard (No-Code) | ✅         |
| SDK                 | ✅         |
| CLI                 | ✅         |
| API                 | ✅         |

</page>

<page url="https://docs.maxun.dev/deep-extraction" title="Deep Extraction: Scrape List & Detail Pages Together">

# Deep Extraction

Deep Extraction lets you capture a list of items and the details from each item's own page in a single robot, automatically.

> **Note:** Deep Extraction is available exclusively in Maxun Cloud.

## What is Deep Extraction?

Here's what it does:

**Without Deep Extraction**
- You visit a list page (like product listings)
- You extract data from that page only
- You manually visit each product to get more details

**With Deep Extraction**
- You capture a list of items (like products)
- You click on one item and capture data from its detail page
- The robot does the same for all other items in the list

## How to Use Deep Extraction

### Step 1: Enable Deep Extraction Before Recording

When creating a new robot:
1. Enter your website URL
2. **Check the "Enable Deep Extraction" checkbox** (this is important!)
3. Choose "Recorder Mode"
4. Click "Start Recording"

![Enable Deep Extraction](deep_extract.png)

> **Important:** You must enable this checkbox before starting your recording. You can't add deep extraction to an existing robot.

### Step 2: Record Your Workflow

Once the recording starts, perform these actions **in order**:

1. **Capture the list**
   - Use "Capture List" to select all items (products, companies, jobs, etc.)
   - Choose what fields you want from each list item

2. **Click on one item**
   - Click on any one item from the list to go to its detail page
   - This shows the robot the pattern to follow

3. **Capture detail page data**
   - Use "Capture Text" or "Capture List" to extract information from the detail page
   - Select all the fields you need

4. **Save your robot**

That's it! When you run this robot, it will:
- Capture all items from the list
- Visit each item's detail page
- Get the detailed information you specified

### Step 3: Run Your Robot

When you run the robot:
- It follows your recorded workflow
- Visits the list page
- Extracts data from the list
- Automatically visits each detail page
- Collects all the information you configured

## When Should You Use Deep Extraction?

Deep Extraction is perfect for:

### E-Commerce Websites
Extract product details from shopping sites:
- **List page**: Product names and prices
- **Detail pages**: Full descriptions, specs, reviews, stock status

### Business Directories
Get complete company information:
- **Directory page**: Company names and locations
- **Company pages**: Full contact details, services, team info

### Job Boards
Capture job listings with all details:
- **Listings page**: Job titles and companies
- **Job pages**: Full descriptions, requirements, salary, how to apply

### Any Multi-Page Data
Whenever you need information that's split across:
- A main list/search/directory page
- Individual detail pages for each item

## What You Need to Know

### Same Website Only
All pages must be on the same website. For example:
- ✅ `example.com/products` → `example.com/products/item-1`
- ❌ `example.com/products` → `different-site.com/details`

### Same Page Structure
All detail pages should look similar:
- Same layout
- Same type of information
- Same element positions

### Works With Pagination
If your list has multiple pages:
- Set up pagination as you normally would
- Deep extraction will process items from all pages

## Example: Extracting Product Information

Let's say you want to extract products from an online store:

### What You Record
1. ✅ Enable Deep Extraction checkbox
2. Visit `shop.example.com/products`
3. Capture List: Product names and prices
4. Click on one product (e.g., "Blue Widget")
5. On the detail page, capture: Full description, specifications, reviews
6. Save robot

### What Happens When You Run It
1. Robot goes to the products page
2. Captures all 50 product names and prices
3. Visits each product's detail page:
   - Blue Widget page
   - Red Gadget page
   - Green Tool page
   - ... and 47 more
4. Gets description, specs, and reviews from each
5. Returns everything in one file

<!-- ## Demo Tutorial
<iframe width="560" height="315" src="https://www.youtube.com/embed/Atr8V5hgiZQ?si=1Qvh0hlLyz3gpZgf" title="Maxun Deep Extract" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> -->

</page>

<page url="https://docs.maxun.dev/extract-login" title="Scrape Data Behind a Login Without Sharing Passwords">

# Extract Behind Login

Maxun lets you extract data that sits behind authenticated (logged-in) pages, such as dashboards, customer portals and member areas, — **without compromising account security**.

We support a **password-free, extension-based login flow on Maxun Cloud**.

---

## Secure, Password-Free Login (Recommended)

You do NOT need to share your username or password with Maxun.

Instead, authentication is handled securely through the <a href="https://chromewebstore.google.com/detail/maxun-session-sync/poljedlhjgoempmkkfahecniimlejgdh">Maxun Chrome Extension</a>, allowing you to extract data behind login using your own active browser session.

### How Password-Free Login Works

1. Install the <a href="https://chromewebstore.google.com/detail/maxun-session-sync/poljedlhjgoempmkkfahecniimlejgdh">Maxun Chrome Extension</a>.
2. Open the extension and connect your Maxun account by adding your API key from the Maxun dashboard.
3. Log in to the target website normally in your browser.
4. Click **Sync** in the extension to share the authenticated session with Maxun Cloud.

### Why This Is Safer

* ✅ No passwords are ever shared or stored
* ✅ Uses your real browser session and cookies
* ✅ Reduces account flagging risk compared to automated logins

> ⚠️ **This extension-based login flow is available exclusively on Maxun Cloud.** Support for 2FA/MFA logins is coming soon.

### Demo: Password-Free Login

<iframe width="560" height="315" src="https://www.youtube.com/embed/brGQOdKbj6E?si=60yrWAEXC3BymhaH" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
---

## OSS Extract Robots: Credential-Based Login

For **OSS (self-hosted) extract robots**, the original login-based approach is still supported.

### How Credential-Based Login Works

1. Enter your username/email and password while training the robot.
2. Perform the required actions to reach the data (see supported actions in [Recorder Mode](/robot/extract/robot-actions)).
3. Credentials entered during training are **encrypted and stored locally**.
4. On each run, the robot automatically logs in using the stored credentials and performs the extraction.

---

## Can my account be flagged or blocked?

Possibly.

We strongly discourage scraping private or restricted data from websites with strict bot-detection policies.

Risks include:

* IP address changes triggering security checks
* Automated login behavior raising flags
* Personal accounts being rate-limited or suspended

### Our Recommendation

* ✅ Use **Maxun Cloud + Chrome Extension** for authenticated extraction
* ⚠️ Avoid automated logins on sensitive websites
* ⚠️ Never use critical personal accounts for scraping

---
### Demo: Credential-Based Login

<iframe width="560" height="315" src="https://www.youtube.com/embed/_-n6OFQI_x4?si=zpnJJQnXQEUM9VwH" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

</page>

<page url="https://docs.maxun.dev/stealth" title="Stealth Mode: Avoid Bot Detection When Scraping">

# Stealth
Stealth mode helps your automations run more reliably on websites that use bot detection or anti-scraping mechanisms.

When enabled, Stealth applies additional techniques to make browser activity appear more human-like and reduce the chances of being blocked.


## Maxun Cloud

- Stealth is enabled by default for paid plans. Free trial does not include stealth.

## Maxun OSS

If you are self-hosting, you can configure your own stealth setup using a <a href="/byop">Bring Your Own Proxy (BYOP)</a> approach.

Please refer to the <a href="/byop">BYOP</a> section for configuration details and recommendations.

## Other Ways to Avoid Getting Blocked

- **Use Maxun Cloud for heavily protected sites.** Cloud includes managed anti-bot infrastructure, automatic proxy rotation and CAPTCHA bypass. See <a href="/cloud-vs-oss">Cloud vs Open Source</a>.
- **Bring your own proxies when self-hosting.** Residential or location-specific proxies reduce blocks. See <a href="/byop">BYOP</a>.
- **Schedule runs at a reasonable frequency.** Very frequent runs can look suspicious, especially for robots that log in.
- **Use dedicated accounts for login robots.** See <a href="/extract-login">Extract Behind Login</a>.

For known limitations such as custom CAPTCHAs and A/B-tested pages, see the <a href="/faq-robot">FAQ</a>.

</page>

<page url="https://docs.maxun.dev/byop" title="BYOP: Bring Your Own Proxy for Web Scraping">

# BYOP (Bring Your Own Proxy)

Maxun provides a feature called BYOP (Bring Your Own Proxy), designed to give users greater control over their data extraction activities.

> **Note:** BYOP is available only in self-hosted installations. With BYOP, you can connect your own proxies to Maxun, enabling:

1. Location-Specific Scraping: Access data specific to certain geographical regions.
2. Geo-Restricted Data Access: Overcome restrictions on location-based content.
3. Enhanced Security: Keep all proxy configurations secure on your self-hosted platform.

## Availability

This feature is available **exclusively for the self-hosted version of Maxun**. For cloud users, Maxun manages all anti-bot infrastructure, ensuring reliable scraping without requiring additional proxy setup.

## Proxy Configuration

HTTP and SOCKS proxies are supported. Currently, proxies are configured per user rather than per robot. However, support for per-robot proxy configuration is coming soon, offering even more granular control over your scraping workflows.

## Authenticated Proxies
You can connect with authenticated proxies that require a username and password. All details are encrypted and securely stored in the database.

If your proxy requires a username and password, always provide them separate from the proxy URL.

### The Right Way

- Proxy URL: http://proxy.com:1337
- Username: myusername
- Password: mypassword

### The Wrong Way
- Proxy URL: http://myusername:mypassword@proxy.com:1337

## Recommended Proxy Providers
Looking for a proxy provider? The following partners offer reliable proxies that work well with Maxun. Maxun users can also take advantage of exclusive discounts.

<table>
<tr>
<td width="20%" align="center" valign="middle">
<a href="https://www.webshare.io/?referral_code=wwhk524ynls2">
<img src="https://github.com/user-attachments/assets/c3a60969-a8f8-4d2c-acef-22bee80c2484" alt="Webshare" width="260" />
</a>
</td>
<td width="80%" valign="middle">
<a href="https://www.webshare.io/?referral_code=wwhk524ynls2"><strong>Webshare</strong></a> gives you instant access to a proxy pool of 80M+ ethically-sourced IPs across 195+ countries, with rotating residential, static ISP, and datacenter options plus a full API. It includes a 100+ Gbps backbone, country/city/state/ZIP/ASN-level targeting, and requires no credit card to start. <strong>Free tier:</strong> 10 proxies + 1GB/month · <strong>Discount:</strong> <code>MAXUN20</code> - 20% off your first purchase.
</td>
</tr>

<tr>
<td width="30%" height="100" align="center" valign="middle">
<a href="https://www.swiftproxy.net/?ref=maxun">
  <img src="https://github.com/user-attachments/assets/a1d2309b-d3bd-45e6-8760-ce1f8dcd18b9" alt="Swiftproxy" width="240" />
</a>
</td>
<td width="80%" valign="middle">
<a href="https://www.swiftproxy.net/?ref=maxun"><strong>Swiftproxy</strong></a> provides 90M+ residential IPs across 220+ locations worldwide, helping Maxun users scrape websites, access geo-restricted content, and collect web data across different regions. Supports rotating and static residential proxies, HTTP(S), and SOCKS5. <strong>Free testing available. </strong><strong>Discount:</strong> <code>PROXY90</code> - 10% off.
</td>
</tr>

<tr>
<td width="20%" align="center" valign="middle">
<a href="https://proxy-seller.com/?partner=224ED2B5F5D24719">
  <img src="https://github.com/user-attachments/assets/ac862778-464d-49ae-af7e-69712d9b74a4" alt="Proxy-Seller" width="300" />
</a>
</td>
<td width="80%" valign="middle">
<a href="https://proxy-seller.com/?partner=224ED2B5F5D24719"><strong>Proxy-Seller</strong></a> gives you reliable proxy infrastructure for any workflow: private residential, mobile (4G/5G/LTE), ISP, and datacenter IPv4/IPv6 proxies, all with HTTP(S) and SOCKS5 support. Tap into 47M+ residential IPs across 220+ locations, target exactly the country, city, or ISP you need, and count on 99.7% uptime backed by 24/7 support.
<strong>Discount:</strong> <code>MAXUNDEV</code> - Get 15% off.
</td>
</tr>

<tr>
<td width="20%" align="center" valign="middle">
<a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=maxun_github_sponsorship">
  <img src="https://github.com/user-attachments/assets/f3f7ea4a-a965-4ff9-aab6-8d257da3b18c" alt="Byteful" width="260"  />
</a>
</td>
<td width="80%" valign="middle">
<a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=maxun_github_sponsorship"><strong>Byteful</strong></a> is a UK-based web data infrastructure platform providing ethically sourced residential, mobile, static residential (ISP) and datacenter proxies alongside API-first tools for web scraping, data collection, and AI-driven automation, processing tens of billions of requests per month. Discount: <code>MAXUN10</code> - Get 10% off Byteful Residential Proxies.
</td>
</tr>

<tr>
<td width="20%" align="center" valign="middle">
<a href="https://mangoproxy.com/?utm_source=github&utm_medium=partner&utm_campaign=getmaxun">
<img src="https://github.com/user-attachments/assets/326b5b8f-6a2c-4aa2-94c8-6a29ae23f2ac" alt="MangoProxy" width="260" />
</a>
</td>
<td width="80%" valign="middle">
<a href="https://mangoproxy.com/?utm_source=github&utm_medium=partner&utm_campaign=getmaxun"><strong>MangoProxy</strong></a>
offers Residential, ISP, Datacenter, and Mobile Proxies in 200+ countries with HTTP(S)/SOCKS5 support and flexible IP management. <strong>Discount:</strong> <code>MAXUN</code> - 8% off Static ISP Proxies.
</td>
</tr>

<tr>
<td width="20%" align="center" valign="middle">
<a href="https://go.nodemaven.com/maxunbyopoct">
<img src="https://github.com/user-attachments/assets/de295d94-61f4-4c97-88f8-cd45cbf3af84" alt="Nodemaven" width="260" />
</a>
</td>
<td width="80%" valign="middle">
<a href="https://go.nodemaven.com/maxunbyopoct"><strong>Nodemaven</strong></a>
offers the most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. Why <a href="https://go.nodemaven.com/maxunbyopoct">Nodemaven</a>? · ZIP targeting · 99.9% uptime · IP filtering: all proxies have fraud score &lt;97%  · No KYC required · Unique free tools: Proxy Bandwidth Checker, Meta Tag Checker, IP Lookup and others! <strong>Discounts:</strong> <code>MAXUN2X</code> — 2X traffic for Black Friday · <code>MAXUN35</code> — 35% off Mobile & Residential · <code>MAXUN40</code> - 40% off ISP (Static)
</td>
</tr>
</table>

</page>

<page url="https://docs.maxun.dev/teams-management" title="Teams: Share Robots, Roles & Permissions">

# Teams Management

Maxun Cloud allows you to collaborate effectively by sharing robots, runs, and settings across multiple accounts using **Teams**. 

Teams provide a unified workspace where members can build, test, and extract data together securely, all without having to share personal account credentials.

![Maxun Teams overview screen](./teams_overview.png)

---

## Setting up a Team

You can quickly create a team within the Maxun Dashboard.

1. Navigate to the **My Teams** panel from the dashboard sidebar.
2. Click **Create Team**.
3. Give your team a descriptive name (e.g., "Marketing Team") and an optional description.

![Create a team in Maxun](./teams_create.png)

> **Note:** Team creation limits are tied to your subscription plan. Users on free or trial tiers may be restricted to their Personal Account and must upgrade to unlock collaborative features. Users are allowed to create 1 team per account.

---

## Team Seat Limits 

The number of members you can invite is determined by the **Team Owner’s Subscription Plan**. 

When a user is invited to a team, the system automatically checks the team owner's plan to evaluate available seats. 

A user seat is considered active if:
- A user has successfully joined the team.
- There is a **Pending Invitation** for a user that has not yet expired.

If the owner's subscription requires renewal or the member limit is exceeded, the associated team will temporarily become paused until the plan is updated or members are removed. 

---

## Roles and Permissions

Maxun provides access control through three specific roles to keep your data secure.

### Admin
- **Full Control**: Create, read, update, and delete resources.
- **Team Management**: Invite new users, remove existing members, and modify member roles.
- **Configurations**: Manage advanced settings and configure API Keys.

### Member
- **Standard Access**: Create, read, and update resources (like building and executing robots).
- **Restrictions**: Cannot delete resources, invite/remove users, or manage API Keys.

### Viewer
- **Read-Only**: View existing resources, robot configurations, and extracted data.
- **Restrictions**: Cannot create, modify, execute, or delete any resources or settings.

![Team roles and permissions in Maxun](./teams_roles.png)

---

## Inviting Members

Admins can easily invite new users to their workspace directly from the dashboard:

1. Inside your team view, click **Invite Member**.
2. Enter the email address of the new user and assign them an appropriate role (`Admin`, `Member`, or `Viewer`).
3. The user will instantly receive an email containing a secure invitation link valid for 7 days.

---

## Switching Contexts (Personal Account vs Teams)

You are never strictly locked into a single workspace. 

By using the **Switch Team** button on your Team configurations page, you can dynamically transition between your private **Personal Account** and any **Teams** you belong to. 

* When in your **Personal Account**, all robots, runs, and data reside securely and privately under your individual account limits.
* When toggled into a **Team Context**, all building, extractions, and API logs will automatically execute and associate directly with that Team, tracking against the team owner's unified account.

![Switch between personal account and team in Maxun](./teams_switch_context.png)

</page>

<page url="https://docs.maxun.dev/integrations/gsheet" title="Scrape Websites to Google Sheets Automatically">

# Google Sheets
> Google Sheets integration is available only for extract robots.

## Overview

Google Sheets integration allows you to automatically sync the data extracted by your robot directly into a Google Sheet. Whenever the robot completes a successful run, the captured data is appended to your designated Google Sheet, enabling seamless and real-time data management.

## Key Features

1. Automatic Data Syncing: Every successful robot run appends the captured data directly to your Google Sheet.
2. Effortless Data Management: Keep all your extracted data organized in a single Google Sheet for easy access and sharing.
3. Real-Time Updates: As soon as a run is complete, the data appears in your Google Sheet.

## Important Note

The data extracted before integrating with Google Sheets will not be synced in the Google Sheet. Only the data extracted after the integration will be synced.

## Setting Up Google Sheets Integration

### Maxun Cloud
#### 1. No Setup Needed
Skip the hassle of creating API keys or Google Cloud projects. Everything is pre-configured for you.

#### 2. One-Click Google Authentication
Simply sign in with your Google account. Maxun Cloud handles the OAuth flow securely in the background.

![Maxun GSheet Integration](gsheet_integration.png)

#### 3. Select a Spreadsheet from Google Drive
Once authenticated, choose any spreadsheet from your Google Drive where you want to store your data.

![Maxun GSheet Sheets](gsheet_sheets.png)

#### 4. Automatic Data Sync
After each successful robot run, your data is automatically appended to the selected sheet. You can disconnect or change the integration at any time.

![Maxun GSheet Integrated](gsheet_integrated.png)

### Maxun Open Source

#### 1. Bring Your Own Google OAuth Keys
Obtain Google OAuth keys by setting up a project in the [Google Cloud Console](https://console.cloud.google.com/).
Ensure the necessary API permissions (Google Sheets and Google Drive APIs) are enabled for the project.

#### 2. Authenticate With Google
Set up the OAuth Keys in the `.env` file and authenticate the robot with google **using the same account that was used to create the Client ID and Client Secret**.

![Maxun GSheet Integration](gsheet_integration.png)

#### 3. Select a Spreadsheet from Google Drive
On successful authentication with google, the robot will be granted access to Google Drive. A spreadsheet can be selected from the Drive to store data from the run.

![Maxun GSheet Sheets](gsheet_sheets.png)

After selecting the Google Sheet, the robot will automatically append data to it after each successful run. If needed, the user can remove the integration at any time.

![Maxun GSheet Integrated](gsheet_integrated.png)

## Demo Tutorial
<iframe width="560" height="315" src="https://www.youtube.com/embed/3EF66DsIze8?si=wuYdtpAhypDLELXU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

</page>

<page url="https://docs.maxun.dev/integrations/airtable" title="Scrape Websites to Airtable Automatically">

# Airtable
> Airtable integration is available only for extract robots.

## Overview

Airtable integration allows you to automatically sync the data extracted by your robot directly into an Airtable Base. Whenever the robot completes a successful run, the captured data is appended to your designated Airtable Base, enabling seamless and real-time data management.


## Key Features

1. Automatic Data Syncing: Every successful robot run appends the captured data directly to your Airtable base.
2. Effortless Data Management: Keep all your extracted data organized in a single Airtable base for easy access and sharing.
3. Real-Time Updates: As soon as a run is complete, the data appears in your Airtable base.

## Important Note

The data extracted before integrating with Airtable will not be synced in the Airtable Base. Only the data extracted after the integration will be synced.

## Setting Up Airtable Integration

### Maxun Cloud
#### 1. Authenticate with Airtable
Simply authenticate using your Airtable account. Maxun Cloud handles the secure OAuth flow automatically.

![Maxun Airtable Integration](airtable_integrate.png)

#### 2. Select Airtable Base & Table
1. On successful Airtable authentication, you can grant access to the Airtable Base the robot should append data to.

![Maxun Airtable Base Access Grant](airtable_access.png)

![Maxun Airtable Fetch Base and Table](airtable_fetch_base.png)

2. After selecting the Airtable base, the robot will automatically append data to it after each successful run. If needed, the user can remove the integration at any time.

### Maxun Open Source

#### 1. Bring Your Own Airtable Client ID
Obtain Airtable Client Id by setting up a project in the [Airtable OAuth Integrations Platform](https://airtable.com/create/oauth).

#### 2. Authenticate with Airtable
Set up the OAuth Keys in the `.env` file and authenticate the robot with airtable **using the same account that was used to create the Client ID**.

![Maxun Airtable Integration](airtable_integrate.png)

#### 3. Select Airtable Base & Table
1. On successful Airtable authentication, you can grant access to the Airtable Base the robot should append data to.
![Maxun Airtable Base Access Grant](airtable_access.png)

![Maxun Airtable Fetch Base and Table](airtable_fetch_base.png)

2. After selecting the Airtable base, the robot will automatically append data to it after each successful run. If needed, the user can remove the integration at any time.


## Demo Tutorial
<iframe width="560" height="315" src="https://www.youtube.com/embed/Yf4Ifi2WuJk?si=9ITe3V3zQePoQDT7" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

</page>

<page url="https://docs.maxun.dev/integrations/n8n" title="n8n Web Scraping Integration: Send Data to n8n">

# n8n

> **Note:** n8n integration is available exclusively in **Maxun Cloud**. This feature is not available in self-hosted installations.

## Overview

n8n integration allows you to automatically send the data extracted by your robot directly to your n8n workflow via webhook. Whenever the robot completes a successful run, the captured data is sent as a POST request to your configured n8n webhook endpoint, enabling seamless automation and data processing within your n8n workflows.

## Key Features

1. Automatic Data Sending: Every successful robot run sends the captured data directly to your n8n webhook endpoint.
2. Real-Time Workflow Triggers: As soon as a run is complete, your n8n workflow is triggered with the fresh data.
3. Flexible Data Processing: Use n8n's powerful workflow capabilities to process, transform, and route your extracted data.

## Important Note

The data extracted before integrating with n8n will not be sent to your workflow. Only the data extracted after the integration will be sent to your n8n webhook.

## Setting Up n8n Integration

### Step 1: Configure Webhook in n8n
Create a webhook trigger node in your n8n workflow and copy the webhook URL.

![n8n Webhook Configuration](n8n_webhook_config.png)

### Step 2: Set Up Integration in Maxun
![n8n Integration Setup](n8n_integration_setup.png)

Configure the webhook endpoint in your robot integration settings with the following fields:

#### Required Fields:
- **Webhook URL**: The n8n webhook endpoint URL (obtained from Step 1)
- **Webhook Name**: A descriptive name for this webhook integration

#### Optional Fields:
- **API Key**: Optional authentication key for additional security
- **n8n Instance URL**: Your n8n instance URL (if using a custom/self-hosted instance)

![n8n Integration Form](n8n_integration_form_configured.png)

### Step 3: Test the Integration
Run your robot to test the webhook integration or use the Test Webhook button. The extracted data will be sent to your n8n workflow.

<!-- ![n8n Integration Success](n8n_integration_success.png) -->

## Data Format

When your robot sends data to n8n, it will be structured as follows:

```json
[
  {
    "headers": {
      "user-agent": "axios/0.26.1",
      "content-type": "application/json",
      "accept": "application/json, text/plain, */*"
    },
    "params": {},
    "query": {},
    "body": {
      "robot_id": "a286c468-d4de-48bf-9555-7aca3df23b30",
      "run_id": "488f4109-fa61-4222-b72c-74e00b17c142",
      "robot_name": "Your Robot Name",
      "timestamp": "2025-08-12T19:46:52.025Z",
      "data": [
        {
          "Field1": "Extracted Value 1",
          "Field2": "Extracted Value 2",
          "Field3": "Extracted Value 3"
        }
      ],
      "metadata": {
        "total_records": 10,
        "data_types": {
          "schema_records": 0,
          "list_records": 10,
          "screenshot_records": 0
        }
      }
    },
    "webhookUrl": "https://your-n8n-instance.com/webhook/your-webhook-id",
    "executionMode": "production"
  }
]
```

## Key Data Fields

- **robot_id**: Unique identifier for the robot
- **run_id**: Unique identifier for this specific run
- **robot_name**: The name of your robot
- **timestamp**: When the data was extracted
- **data**: Array containing all extracted records
- **metadata**: Information about the extraction results

## Demo Tutorial

In this tutorial, learn how to use Maxun with n8n and GPT to generate responses for Chrome extension reviews automatically!

<iframe width="560" height="315" src="https://www.youtube.com/embed/0o2xMOa8Ee8?si=lnZE-Idky-X46hd-" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

</page>

<page url="https://docs.maxun.dev/integrations/claude-code" title="Claude Code Web Scraping Skill">

# Claude Code

The Maxun skill for <a href="https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview" target="_blank">Claude Code</a> lets you list, run, and retrieve results from your Maxun robots directly inside your terminal workflow. 

---

## Prerequisites

- <a href="https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview" target="_blank">Claude Code</a> installed
- A Maxun Cloud account at <a href="https://app.maxun.dev" target="_blank">app.maxun.dev</a> or a self-hosted instance
- API Key from <a href="/api/api">Maxun Dashboard</a>

---

## Setup

### 1. Identify your working directory

Claude Code keeps its config file recursively up from the directory where you run the tool. For instance, you could use `~/.claude/skills/maxun`.

### 2. Download the skill files

```bash
# 1. Create the skill directory
mkdir -p ~/.claude/skills/maxun/scripts

# 2. Download the skill files
curl -sf -o ~/.claude/skills/maxun/SKILL.md \
  https://raw.githubusercontent.com/getmaxun/maxun-skills/main/claude-code/.claude/skills/maxun/SKILL.md
curl -sf -o ~/.claude/skills/maxun/scripts/maxun.sh \
  https://raw.githubusercontent.com/getmaxun/maxun-skills/main/claude-code/.claude/skills/maxun/scripts/maxun.sh

# 3. Make the script executable
chmod +x ~/.claude/skills/maxun/scripts/maxun.sh
```

### 3. Configure your API key

The skill automatically looks for a `.env` file in your **project root** and loads the variables without any extra configuration needed inside Claude Code.

Just add to your project's `.env`:
```bash
MAXUN_API_KEY="your_api_key_here"
```

For **self-hosted Maxun** users, also set your base URL in the same `.env` file:
```bash
MAXUN_BASE_URL="http://localhost:8080"
```

*(Alternatively, you can export these variables directly in your bash or zsh profiles if you prefer them to be globally injected without a `.env` file).*

---

## Usage

Once installed, your agent will automatically understand requests about your robots directly in your command line:

- "What Maxun robots do I have?"
- "Run my product scraper robot"
- "Show me the result for the last run of robot abc123"
- "Abort run run_xyz789 for robot abc123"

### Available Commands (Slash Commands)

| Command / Prompt | Action |
| --- | --- |
| `/maxun list` | List all robots |
| `/maxun run <robotId>` | Run a robot (synchronous) |
| `/maxun result <robotId> <runId>` | Fetch a specific run result |
| `/maxun runs <robotId>` | List all past runs for a robot |
| `/maxun get <robotId>` | Get detailed robot metadata |
| `/maxun abort <robotId> <runId>` | Stop an in-progress execution |

---

## Resources

- <a href="https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview" target="_blank">Claude Code</a>
- <a href="https://github.com/getmaxun/maxun-skills" target="_blank">Maxun Skills Repository</a>

</page>

<page url="https://docs.maxun.dev/integrations/openclaw" title="OpenClaw Web Scraping Skill">

# OpenClaw

The Maxun skill for <a href="https://openclaw.ai" target="_blank">OpenClaw</a> lets you list, run, and retrieve results from your Maxun robots — through any messaging app connected to your OpenClaw gateway (WhatsApp, Telegram, Discord, and more).

The skill is published on <a href="https://clawhub.ai/RohitR311/maxun" target="_blank">ClawHub</a>, the OpenClaw skill registry.

---

## Prerequisites

- OpenClaw running (self-hosted gateway)
- A Maxun Cloud account at <a href="https://app.maxun.dev" target="_blank">app.maxun.dev</a> or a self-hosted instance
- API Key from <a href="/api/api">Maxun Dashboard</a>
- The ClawHub CLI installed

```bash
npm i -g clawhub
```

---

## Setup

### 1. Install the skill

```bash
clawhub install maxun
```

This downloads the skill into your `./skills` directory and records it in `.clawhub/lock.json`.

### 2. Configure your API key

The skill automatically looks for a `.env` file in your OpenClaw gateway directory and loads the variables seamlessly without needing to export them manually. Create or edit a `.env` file in the root of your OpenClaw deployment:

```bash
MAXUN_API_KEY="your_api_key_here"
```

For **self-hosted Maxun** users, also set your base URL in the same `.env` file:
```bash
MAXUN_BASE_URL="http://localhost:8080"
```

*(Alternatively, you can configure your API key directly in your OpenClaw `~/.openclaw/openclaw.json` under the skill's `env` block or run `openclaw config set env.MAXUN_API_KEY "your_key"`)*

### 3. Restart the gateway

```bash
openclaw restart
```

The skill is loaded on restart and is immediately available from any connected app.

---

## Usage

Once installed, send a message from any connected app — OpenClaw automatically detects when your request involves Maxun robots and handles it. You can also invoke the skill directly with `/maxun`.

### Listing Robots

```
"What Maxun robots do I have?"
"Show me all my robots"
"List my scraping robots"
```

### Running a Robot

```
"Run my Amazon price scraper"
"Execute the product catalog robot"
"Run robot abc123 and show me the results"
```

:::tip
You don't need to know the robot ID. Describe it by name — the skill resolves it automatically before running.
:::

### Getting Results

```
"Show me the results from my last run"
"What did the price scraper extract yesterday?"
"Get the output from run xyz789"
```

---

## Common Workflows

### First Time

```
"What robots do I have?"       → lists all robots
"Tell me about the jobs robot" → shows details
"Run it"                       → executes and returns results
```

### Chained Requests

OpenClaw handles multi-step prompts in one message:

```
"List my robots, run the most recently created one, and summarize what it found"
```

---

## Model Recommendation

:::tip
For the best results, configure OpenClaw to use a capable model from providers like **Anthropic** or **OpenAI**. Weaker models may struggle to reliably trigger the skill, resolve robot names, or interpret extracted data correctly. A stronger model noticeably improves skill invocation accuracy and response quality.
:::

---

## Resources

- <a href="https://openclaw.ai" target="_blank">OpenClaw</a>
- <a href="https://clawhub.ai/RohitR311/maxun" target="_blank">Maxun skill on ClawHub</a>
- <a href="https://docs.openclaw.ai/tools/skills" target="_blank">OpenClaw Skills Documentation</a>

</page>

<page url="https://docs.maxun.dev/integrations/langchain" title="LangChain Web Scraping Tool: Maxun Integration">

# LangChain

Build LLM chains and agents with web scraping capabilities using Maxun and LangChain.

## Setup

Install dependencies:

```bash
npm install maxun-sdk @langchain/openai @langchain/core zod
```

Create `.env` file:

```env
MAXUN_API_KEY=your_maxun_key
OPENAI_API_KEY=your_openai_key
```

Initialize clients:

```javascript

const maxun = {
  scrape: new Scrape({ apiKey: process.env.MAXUN_API_KEY }),
  search: new Search({ apiKey: process.env.MAXUN_API_KEY })
};

const model = new ChatOpenAI({
  model: 'gpt-5-mini',
  apiKey: process.env.OPENAI_API_KEY
});
```

## Scrape + Chat

Scrape a website and chat about its content:

```javascript
async function scrapeAndChat() {
  const robot = await maxun.scrape.create('Scraper', 'https://medium.com', {
    formats: ['markdown']
  });
  const result = await robot.run();

  const response = await model.invoke([
    { role: 'user', content: `Summarize the trending topics in 3 bullet points: ${result.data.markdown?.slice(0, 5000)}` }
  ]);

  console.log('Summary:', response.content);
}
```

## Chains

Use LangChain chains for structured processing:

```javascript
async function useChains() {
  const robot = await maxun.scrape.create('Scraper', 'https://maxun.dev', {
    formats: ['markdown']
  });
  const result = await robot.run();

  const prompt = ChatPromptTemplate.fromMessages([
    ['system', 'You are an expert at analyzing company websites. Be concise.'],
    ['user', 'Extract the company name and main features from: {content}']
  ]);

  const chain = prompt.pipe(model).pipe(new StringOutputParser());

  const analysis = await chain.invoke({
    content: result.data.markdown?.slice(0, 5000)
  });

  console.log('Analysis:', analysis);
}
```

## Tool Calling

Let the LLM decide when to scrape:

```javascript
async function toolCalling() {
  const scrapeTool = new DynamicStructuredTool({
    name: 'scrape_website',
    description: 'Scrape content from any website URL',
    schema: z.object({
      url: z.string().url().describe('The URL to scrape')
    }),
    func: async ({ url }) => {
      const robot = await maxun.scrape.create('Scraper', url, {
        formats: ['markdown']
      });
      const result = await robot.run();
      return result.data.markdown?.slice(0, 3000) || 'No content scraped';
    }
  });

  const modelWithTools = model.bindTools([scrapeTool]);

  const response = await modelWithTools.invoke(
    'What does Stripe do? Visit stripe.com and tell me about it.'
  );

  console.log('Response:', response.content);
  console.log('Tool calls:', response.tool_calls);
}
```

## Search + Analyze

Search the web and analyze results:

```javascript
async function searchAndAnalyze() {
  const robot = await maxun.search.create('Searcher', {
    query: 'best places to visit in Europe',
    mode: 'discover',
    limit: 5
  });
  const result = await robot.run();

  const searchData = result.data.searchData as any;
  const searchResults = searchData ? Object.values(searchData)[0] as any : null;
  const results = searchResults?.results || [];

  const context = results
    .map((r: any) => `${r.title}: ${r.description}`)
    .join('\n');

  const response = await model.invoke([
    { role: 'user', content: `Analyze these travel recommendations and list the top 5 destinations:\n${context}` }
  ]);

  console.log('Analysis:', response.content);
}
```

## Resources

- [LangChain Documentation](https://js.langchain.com/docs/)

</page>

<page url="https://docs.maxun.dev/integrations/langgraph" title="LangGraph Web Scraping: Maxun Integration">

# LangGraph

Build stateful, multi-step AI workflows with web scraping using Maxun and LangGraph.

## Setup

Install dependencies:

```bash
npm install maxun-sdk @langchain/langgraph @langchain/openai @langchain/core
```

Create `.env` file:

```env
MAXUN_API_KEY=your_maxun_key
OPENAI_API_KEY=your_openai_key
```

Initialize clients:

```javascript

const maxun = {
  scrape: new Scrape({ apiKey: process.env.MAXUN_API_KEY }),
  search: new Search({ apiKey: process.env.MAXUN_API_KEY })
};

const llm = new ChatOpenAI({
  model: 'gpt-5-mini',
  apiKey: process.env.OPENAI_API_KEY
});
```

## Basic Workflow

Scrape a website and analyze with LLM:

```javascript
async function basicWorkflow() {
  async function scrapeNode(state: typeof MessagesAnnotation.State) {
    const robot = await maxun.scrape.create('Scraper', 'https://maxun.dev', {
      formats: ['markdown']
    });
    const result = await robot.run();
    const content = result.data.markdown?.slice(0, 3000) || 'No content found';

    return {
      messages: [new SystemMessage(`Here is the webpage content to analyze:\n\n${content}`)]
    };
  }

  async function analyzeNode(state: typeof MessagesAnnotation.State) {
    const messages = state.messages;
    const systemMsg = messages.find(m => m instanceof SystemMessage);
    const humanMsg = messages.find(m => m instanceof HumanMessage);

    const prompt = `Context:\n${systemMsg?.content || ''}\n\nRequest:\n${humanMsg?.content || 'Summarize the content'}`;

    const response = await llm.invoke([new HumanMessage(prompt)]);
    return { messages: [response] };
  }

  const graph = new StateGraph(MessagesAnnotation)
    .addNode('scrape', scrapeNode)
    .addNode('analyze', analyzeNode)
    .addEdge(START, 'scrape')
    .addEdge('scrape', 'analyze')
    .addEdge('analyze', END);

  const app = graph.compile();

  const result = await app.invoke({
    messages: [new HumanMessage('What are the main features of this product?')]
  });

  const lastMessage = result.messages[result.messages.length - 1];
  console.log('Result:', lastMessage.content);
}
```

## Multi-Step Workflow

Search for URLs, scrape them, and summarize:

```javascript
const WorkflowState = Annotation.Root({
  urls: Annotation<string[]>({
    reducer: (_, y) => y,
    default: () => []
  }),
  scrapedData: Annotation<Array<{ url: string; content: string }>>({
    reducer: (_, y) => y,
    default: () => []
  }),
  summary: Annotation<string>({
    reducer: (_, y) => y,
    default: () => ''
  })
});

async function multiStepWorkflow() {
  async function searchNode(state: typeof WorkflowState.State) {
    const robot = await maxun.search.create('Searcher', {
      query: 'best places to visit in Europe',
      mode: 'discover',
      limit: 3
    });
    const result = await robot.run();

    const searchData = result.data.searchData as any;
    const searchResults = searchData ? Object.values(searchData)[0] as any : null;
    const urls = searchResults?.results?.map((r: any) => r.url).filter(Boolean) || [];

    return { urls };
  }

  async function scrapeMultiple(state: typeof WorkflowState.State) {
    const scrapedData = [];

    for (const url of state.urls.slice(0, 2)) {
      try {
        const robot = await maxun.scrape.create('Scraper', url, {
          formats: ['markdown']
        });
        const result = await robot.run();
        scrapedData.push({ url, content: result.data.markdown?.slice(0, 2000) || '' });
      } catch (e) {
        console.log(`Failed: ${url}`);
      }
    }

    return { scrapedData };
  }

  async function summarizeAll(state: typeof WorkflowState.State) {
    const combinedContent = state.scrapedData
      .map(item => `Source: ${item.url}\n${item.content}`)
      .join('\n\n---\n\n');

    const response = await llm.invoke([
      { role: 'user', content: `Summarize the best travel destinations from these sources:\n${combinedContent}` }
    ]);

    return { summary: response.content as string };
  }

  const workflow = new StateGraph(WorkflowState)
    .addNode('search', searchNode)
    .addNode('scrape', scrapeMultiple)
    .addNode('summarize', summarizeAll)
    .addEdge(START, 'search')
    .addEdge('search', 'scrape')
    .addEdge('scrape', 'summarize')
    .addEdge('summarize', END);

  const app = workflow.compile();

  const result = await app.invoke({ urls: [] });
  console.log('Summary:', result.summary);
}
```

## Resources

- [LangGraph Documentation](https://langchain-ai.github.io/langgraph/)

</page>

<page url="https://docs.maxun.dev/integrations/llamaindex" title="LlamaIndex RAG With Live Web Data">

# LlamaIndex

Build RAG (Retrieval-Augmented Generation) applications with web scraping using Maxun and LlamaIndex.

## Setup

Install dependencies:

```bash
npm install maxun-sdk llamaindex @llamaindex/openai
```

Create `.env` file:

```env
MAXUN_API_KEY=your_maxun_key
OPENAI_API_KEY=your_openai_key
```

Initialize clients:

```javascript

Settings.llm = new OpenAI({ model: 'gpt-5-mini' });
Settings.embedModel = new OpenAIEmbedding({ model: 'text-embedding-3-small' });

const maxun = {
  scrape: new Scrape({ apiKey: process.env.MAXUN_API_KEY }),
  search: new Search({ apiKey: process.env.MAXUN_API_KEY })
};
```

## RAG with Single Page

Scrape a website and answer questions about it:

```javascript
async function ragSinglePage() {
  const robot = await maxun.scrape.create('Scraper', 'https://www.ycombinator.com/jobs', {
    formats: ['markdown']
  });
  const result = await robot.run();

  const document = new Document({
    text: result.data.markdown?.slice(0, 5000) || '',
    id_: 'ycombinator-jobs',
    metadata: { url: 'https://www.ycombinator.com/jobs' }
  });

  const index = await VectorStoreIndex.fromDocuments([document]);

  const queryEngine = index.asQueryEngine();
  const response = await queryEngine.query({
    query: 'What job opportunities are available at YC startups?'
  });

  console.log('Answer:', response.toString());
}
```

## RAG with Multiple Pages

Search for pages, scrape them, and build a knowledge base:

```javascript
async function ragMultiplePages() {
  const searchRobot = await maxun.search.create('Searcher', {
    query: 'best places to visit in Europe',
    mode: 'discover',
    limit: 3
  });
  const searchResult = await searchRobot.run();

  const searchData = searchResult.data.searchData as any;
  const searchResults = searchData ? Object.values(searchData)[0] as any : null;
  const urls = searchResults?.results?.map((r: any) => r.url).filter(Boolean) || [];

  const documents: Document[] = [];
  for (const url of urls.slice(0, 2)) {
    try {
      const robot = await maxun.scrape.create('Scraper', url, {
        formats: ['markdown']
      });
      const result = await robot.run();

      documents.push(new Document({
        text: result.data.markdown?.slice(0, 3000) || '',
        id_: url,
        metadata: { url }
      }));
    } catch (e) {
      console.log(`Failed: ${url}`);
    }
  }

  const index = await VectorStoreIndex.fromDocuments(documents);

  const queryEngine = index.asQueryEngine();
  const response = await queryEngine.query({
    query: 'What are the top recommended destinations in Europe and why?'
  });

  console.log('Answer:', response.toString());
}
```

## Resources

- [LlamaIndex Documentation](https://ts.llamaindex.ai/)

</page>

<page url="https://docs.maxun.dev/integrations/mastra" title="Mastra Web Scraping Workflows With Maxun">

# Mastra

Build AI agent workflows with web scraping using Maxun and Mastra.

## Setup

Install dependencies:

```bash
npm install maxun-sdk @mastra/core zod
```

Create `.env` file:

```env
MAXUN_API_KEY=your_maxun_key
OPENAI_API_KEY=your_openai_key
```

Initialize clients:

```javascript

const maxun = {
  scrape: new Scrape({ apiKey: process.env.MAXUN_API_KEY }),
  search: new Search({ apiKey: process.env.MAXUN_API_KEY })
};

const agent = new Agent({
  name: 'summarizer',
  instructions: 'You are a helpful assistant that creates concise summaries.',
  model: {
    id: 'openai/gpt-5-mini',
    apiKey: process.env.OPENAI_API_KEY
  }
});
```

## Multi-Step Workflow

Search, scrape, and summarize:

```javascript
const searchStep = createStep({
  id: 'search',
  inputSchema: z.object({
    query: z.string()
  }),
  outputSchema: z.object({
    url: z.string(),
    title: z.string()
  }),
  execute: async ({ inputData }) => {
    const robot = await maxun.search.create('Searcher', {
      query: inputData.query,
      mode: 'discover',
      limit: 1
    });
    const result = await robot.run();

    const searchData = result.data.searchData as any;
    const searchResults = searchData ? Object.values(searchData)[0] as any : null;
    const firstResult = searchResults?.results?.[0];

    if (!firstResult) {
      throw new Error('No search results found');
    }

    return {
      url: firstResult.url,
      title: firstResult.title
    };
  }
});

const scrapeStep = createStep({
  id: 'scrape',
  inputSchema: z.object({
    url: z.string(),
    title: z.string()
  }),
  outputSchema: z.object({
    markdown: z.string(),
    title: z.string()
  }),
  execute: async ({ inputData }) => {
    const robot = await maxun.scrape.create('Scraper', inputData.url, {
      formats: ['markdown']
    });
    const result = await robot.run();

    return {
      markdown: result.data.markdown?.slice(0, 3000) || '',
      title: inputData.title
    };
  }
});

const summarizeStep = createStep({
  id: 'summarize',
  inputSchema: z.object({
    markdown: z.string(),
    title: z.string()
  }),
  outputSchema: z.object({
    summary: z.string()
  }),
  execute: async ({ inputData }) => {
    const prompt = `Summarize in 2-3 sentences:\n\nTitle: ${inputData.title}\n\n${inputData.markdown}`;
    const result = await agent.generate(prompt);

    return { summary: result.text };
  }
});

const workflow = createWorkflow({
  id: 'search-scrape-summarize',
  inputSchema: z.object({
    query: z.string()
  }),
  outputSchema: z.object({
    summary: z.string()
  }),
  steps: [searchStep, scrapeStep, summarizeStep]
})
  .then(searchStep)
  .then(scrapeStep)
  .then(summarizeStep)
  .commit();
```

## Running the Workflow

```javascript
async function main() {
  const run = await workflow.createRunAsync();
  const result = await run.start({
    inputData: { query: 'best places to visit in Europe' }
  });

  if (result.status === 'success') {
    const { summarize } = result.steps;
    if (summarize.status === 'success') {
      console.log('Summary:', summarize.output.summary);
    }
  } else {
    console.error('Workflow failed:', result.status);
  }
}
```

## Resources

- [Mastra Documentation](https://mastra.ai/docs)

</page>

<page url="https://docs.maxun.dev/integrations/openai" title="OpenAI Function Calling for Web Scraping">

# OpenAI SDK

Use OpenAI's function calling with Maxun for autonomous web scraping.

## Setup

Install dependencies:

```bash
npm install maxun-sdk openai
```

Create `.env` file:

```env
MAXUN_API_KEY=your_maxun_key
OPENAI_API_KEY=your_openai_key
```

Initialize clients:

```javascript

const maxun = {
  scrape: new Scrape({ apiKey: process.env.MAXUN_API_KEY }),
  search: new Search({ apiKey: process.env.MAXUN_API_KEY })
};

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const MODEL = 'gpt-5-mini';
```

## Scrape + Summarize

Scrape a website and summarize with LLM:

```javascript
async function scrapeAndSummarize() {
  const robot = await maxun.scrape.create('Scraper', 'https://medium.com', {
    formats: ['markdown']
  });
  const result = await robot.run();

  const completion = await openai.chat.completions.create({
    model: MODEL,
    messages: [
      { role: 'user', content: `Summarize the trending topics in 3 bullet points: ${result.data.markdown?.slice(0, 5000)}` }
    ]
  });

  console.log('Summary:', completion.choices[0]?.message.content);
}
```

## Function Calling

Let LLM call Maxun tools automatically:

```javascript
async function functionCalling() {
  const tools: OpenAI.Chat.Completions.ChatCompletionTool[] = [{
    type: 'function',
    function: {
      name: 'scrape_website',
      description: 'Scrape content from any website URL',
      parameters: {
        type: 'object',
        properties: {
          url: { type: 'string', description: 'The URL to scrape' }
        },
        required: ['url']
      }
    }
  }];

  const response = await openai.chat.completions.create({
    model: MODEL,
    messages: [{ role: 'user', content: 'What does Maxun do? Visit maxun.dev and tell me.' }],
    tools
  });

  const message = response.choices[0]?.message;

  if (message?.tool_calls && message.tool_calls.length > 0) {
    for (const toolCall of message.tool_calls) {
      if (toolCall.type !== 'function') continue;

      const args = JSON.parse(toolCall.function.arguments);

      const robot = await maxun.scrape.create('Scraper', args.url, {
        formats: ['markdown']
      });
      const result = await robot.run();

      const finalResponse = await openai.chat.completions.create({
        model: MODEL,
        messages: [
          { role: 'user', content: 'What does Maxun do? Visit maxun.dev and tell me.' },
          message,
          {
            role: 'tool',
            tool_call_id: toolCall.id,
            content: result.data.markdown?.slice(0, 3000) || 'No content'
          }
        ]
      });

      console.log('Response:', finalResponse.choices[0]?.message?.content);
    }
  } else {
    console.log('Direct response:', message?.content);
  }
}
```

## Search + Analyze

Search the web and analyze results:

```javascript
async function searchAndAnalyze() {
  const robot = await maxun.search.create('Searcher', {
    query: 'best places to visit in Europe',
    mode: 'discover',
    limit: 5
  });
  const result = await robot.run();

  const searchData = result.data.searchData as any;
  const searchResults = searchData ? Object.values(searchData)[0] as any : null;
  const results = searchResults?.results || [];

  const context = results
    .map((r: any) => `${r.title}: ${r.description}`)
    .join('\n');

  const completion = await openai.chat.completions.create({
    model: MODEL,
    messages: [
      { role: 'user', content: `Analyze these travel recommendations and list the top 5 destinations:\n${context}` }
    ]
  });

  console.log('Analysis:', completion.choices[0]?.message?.content);
}
```

## Resources

- [OpenAI API Reference](https://platform.openai.com/docs/api-reference)
- [Function Calling Guide](https://platform.openai.com/docs/guides/function-calling)

</page>

<page url="https://docs.maxun.dev/integrations/vercel-ai-sdk" title="Vercel AI SDK Web Scraping Tools">

# Vercel AI SDK

Build AI-powered web applications with scraping capabilities using Maxun and the Vercel AI SDK.

## Setup

Install dependencies:

```bash
npm install maxun-sdk ai @ai-sdk/openai zod
```

Create `.env` file:

```env
MAXUN_API_KEY=your_maxun_key
OPENAI_API_KEY=your_openai_key
```

Initialize clients:

```javascript

const maxun = {
  scrape: new Scrape({ apiKey: process.env.MAXUN_API_KEY }),
  search: new Search({ apiKey: process.env.MAXUN_API_KEY }),
  crawl: new Crawl({ apiKey: process.env.MAXUN_API_KEY }),
  extract: new Extract({ apiKey: process.env.MAXUN_API_KEY })
};

const model = openai('gpt-5-mini');
```

## Defining Tools

```javascript
const scrapeTool = tool({
  description: 'Scrape content from a website URL',
  parameters: z.object({
    url: z.string().url().describe('The URL to scrape')
  }),
  execute: async ({ url }) => {
    const robot = await maxun.scrape.create('Scraper', url, { formats: ['markdown'] });
    const result = await robot.run();
    return result.data.markdown?.slice(0, 3000) || 'No content';
  }
});

const searchTool = tool({
  description: 'Search the web for information',
  parameters: z.object({
    query: z.string().describe('Search query')
  }),
  execute: async ({ query }) => {
    const robot = await maxun.search.create('Searcher', { query, mode: 'discover', limit: 5 });
    const result = await robot.run();
    const searchData = result.data.searchData as any;
    const searchResults = searchData ? Object.values(searchData)[0] as any : null;
    return searchResults?.results || [];
  }
});

const crawlTool = tool({
  description: 'Crawl a website to discover pages',
  parameters: z.object({
    url: z.string().url().describe('The URL to crawl')
  }),
  execute: async ({ url }) => {
    const robot = await maxun.crawl.create('Crawler', url, { mode: 'domain', limit: 5 });
    const result = await robot.run();
    return result.data.crawlData || [];
  }
});

const extractTool = tool({
  description: 'Extract structured data from a website',
  parameters: z.object({
    url: z.string().url().describe('The URL to extract from'),
    prompt: z.string().describe('What to extract')
  }),
  execute: async ({ url, prompt }) => {
    const robot = await maxun.extract.extract({ url, prompt, llmProvider: 'openai' });
    const result = await robot.run();
    return result.data.listData || result.data.textData || {};
  }
});
```

## Scrape

```javascript
async function scrapeExample() {
  const { text } = await generateText({
    model,
    prompt: 'Scrape https://stripe.com and summarize what the company does in 3 bullet points',
    tools: { scrape: scrapeTool },
    maxSteps: 3
  });

  console.log(text);
}
```

## Search

```javascript
async function searchExample() {
  const { text } = await generateText({
    model,
    prompt: 'Search for best places to visit in Europe and recommend the top 5 destinations',
    tools: { search: searchTool },
    maxSteps: 3
  });

  console.log(text);
}
```

## Crawl

```javascript
async function crawlExample() {
  const { text } = await generateText({
    model,
    prompt: 'Crawl https://stripe.com and list the main sections of the website',
    tools: { crawl: crawlTool },
    maxSteps: 3
  });

  console.log(text);
}
```

## Extract

```javascript
async function extractExample() {
  const { text } = await generateText({
    model,
    prompt: 'Extract book titles and prices from https://books.toscrape.com',
    tools: { extract: extractTool },
    maxSteps: 3
  });

  console.log(text);
}
```

## Resources

- [Vercel AI SDK Documentation](https://sdk.vercel.ai/)

</page>

<page url="https://docs.maxun.dev/ai" title="Build With AI: Maxun for AI Agents & LLM Apps">

# Build With AI

Maxun gives AI agents and LLM apps clean, live web data. Agents can scrape a page into Markdown, crawl a site, search the web, or describe the data they need and let Maxun extract it, then run the same robot again whenever they need fresh data.

There are four ways to use Maxun with AI:

| | What it is | Best for |
|---|---|---|
| **[MCP server](/mcp/setup)** | Maxun's tools inside any MCP client | Claude, Cursor, Windsurf, VS Code, Codex and other agents that need web data while they work |
| **[Agent skills](/integrations/claude-code)** | Ready-made skills that run your Maxun robots | Claude Code and OpenClaw users who already have robots |
| **[LLM-ready docs](/ai/llm-ready-docs)** | `llms.txt`, `llms-full.txt` and every page as Markdown | Coding agents that write Maxun code for you |
| **[AI frameworks](#ai-frameworks)** | The Maxun SDK as a tool or loader in your framework | LangChain, LangGraph, LlamaIndex, Mastra, OpenAI and Vercel AI SDK apps |

## Quick start: let your agent set it up

Copy this prompt into Claude Code, Cursor, Codex or any coding agent. It reads the docs and adds Maxun to your project.

<PromptCard title="Add Maxun to my project">{`Add Maxun (https://maxun.dev) to this project so it can scrape, crawl, search and extract web data.

1. Read https://docs.maxun.dev/llms.txt to learn how Maxun works. Fetch any page as Markdown by adding .md to its URL.
2. Pick the official SDK for this project's language: maxun-sdk for Node.js/TypeScript, or maxun for Python.
3. Install it and read the API key from the MAXUN_API_KEY environment variable. Never hard-code the key. If .env.example exists, add MAXUN_API_KEY to it.
4. Add a small example that scrapes https://maxun.dev into Markdown and prints the result.
5. Tell me where to get an API key: https://app.maxun.dev, under Settings → API Key.`}</PromptCard>

To give your agent Maxun's tools directly, follow [Add Maxun to your agent](/ai/add-to-agent).

## Connect your agent with MCP

The [Maxun MCP server](/mcp/setup) works with any MCP client. On Maxun Cloud it is one URL and an API key:

```text
URL:     https://app.maxun.dev/api/mcp
Header:  x-api-key: <your-api-key>
```

For example, in Claude Code:

```bash
claude mcp add --transport http maxun https://app.maxun.dev/api/mcp --header "x-api-key: your_api_key_here"
```

Your agent then gets these [tools](/mcp/tools):

| Tool | What it does |
|---|---|
| `maxun_scrape` | Turn a page into Markdown, HTML, text, links, a summary or screenshots, and answer questions about it |
| `maxun_crawl` | Crawl a website and return the content of every page |
| `maxun_search` | Search the web and optionally scrape each result |
| `maxun_agent` | Describe the data you want and let Maxun's AI build and run an extraction robot |
| `maxun_list_robots`, `maxun_run_robot`, ... | Run and inspect the robots you already have |

See [Add Maxun to your agent](/ai/add-to-agent) for Cursor, VS Code, Windsurf, Claude Desktop, Codex and Gemini CLI.

## AI frameworks

Use the Maxun SDK as a tool or document loader in your framework of choice:

| Framework | Guide |
|---|---|
| LangChain | [LangChain](/integrations/langchain) |
| LangGraph | [LangGraph](/integrations/langgraph) |
| LlamaIndex | [LlamaIndex](/integrations/llamaindex) |
| Mastra | [Mastra](/integrations/mastra) |
| OpenAI SDK (function calling) | [OpenAI](/integrations/openai) |
| Vercel AI SDK | [Vercel AI SDK](/integrations/vercel-ai-sdk) |

## Writing prompts for extraction

[AI mode](/robot/extract/llm-extraction), the `maxun_agent` tool and the SDK's `extract()` all take a plain-English description of the data you want. See [How to write prompts](/llm-prompts) for examples that work well.

</page>

<page url="https://docs.maxun.dev/ai/add-to-agent" title="Add Maxun to Your AI Agent: Claude, Cursor, Codex & More">

# Add Maxun to Your Agent

Give your AI agent the ability to scrape, crawl, search and extract web data in a couple of minutes. There are three options:

1. **[MCP server](#connect-with-mcp)** (recommended): adds Maxun's tools to any MCP client.
2. **[Agent skill](#install-an-agent-skill)**: a skill for Claude Code or OpenClaw that runs your existing robots.
3. **[Docs in context](#give-your-agent-the-docs)**: point a coding agent at the docs so it writes Maxun code for you.

## Before you start

Get an API key from the [Maxun dashboard](https://app.maxun.dev) under **Settings → API Key**. See [API Key](/api/api) for details. Replace `your_api_key_here` below with your key.

## Connect with MCP

The examples below use **Maxun Cloud** (`https://app.maxun.dev/api/mcp`). If you self-host Maxun, follow the [self-hosted MCP setup](/mcp/setup#self-hosted-oss) instead.

<Tabs groupId="mcp-client">
<TabItem value="claude-code" label="Claude Code" default>

Run this in your terminal:

```bash
claude mcp add --transport http maxun https://app.maxun.dev/api/mcp --header "x-api-key: your_api_key_here"
```

Add `--scope user` to make Maxun available in every project, or `--scope project` to share it with your team through `.mcp.json`. Run `/mcp` inside Claude Code to check the connection.

</TabItem>
<TabItem value="cursor" label="Cursor">

Add Maxun to `~/.cursor/mcp.json` (all projects) or `.cursor/mcp.json` (this project):

```json
{
  "mcpServers": {
    "maxun": {
      "url": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here"
      }
    }
  }
}
```

Open **Cursor Settings → MCP** and check that `maxun` shows a green dot.

</TabItem>
<TabItem value="vscode" label="VS Code">

Add Maxun to `.vscode/mcp.json` in your workspace:

```json
{
  "servers": {
    "maxun": {
      "type": "http",
      "url": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here"
      }
    }
  }
}
```

Then open Copilot Chat in **Agent** mode and select the Maxun tools.

</TabItem>
<TabItem value="windsurf" label="Windsurf">

Add Maxun to `~/.codeium/windsurf/mcp_config.json`:

```json
{
  "mcpServers": {
    "maxun": {
      "serverUrl": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here"
      }
    }
  }
}
```

Refresh the MCP servers in Windsurf's Cascade panel.

</TabItem>
<TabItem value="claude-desktop" label="Claude Desktop">

Open **Settings → Developer → Edit Config** and add Maxun to `claude_desktop_config.json`. Claude Desktop uses [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) to connect, so Node.js must be installed.

```json
{
  "mcpServers": {
    "maxun": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://app.maxun.dev/api/mcp",
        "--header",
        "x-api-key:your_api_key_here"
      ]
    }
  }
}
```

Restart Claude Desktop. The Maxun tools appear under the tools icon in the chat box.

</TabItem>
<TabItem value="codex" label="Codex">

Add Maxun to `~/.codex/config.toml`:

```toml
[mcp_servers.maxun]
command = "npx"
args = ["-y", "mcp-remote", "https://app.maxun.dev/api/mcp", "--header", "x-api-key:your_api_key_here"]
```

Start a new Codex session to load the server.

</TabItem>
<TabItem value="gemini" label="Gemini CLI">

Add Maxun to `~/.gemini/settings.json`:

```json
{
  "mcpServers": {
    "maxun": {
      "httpUrl": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here"
      }
    }
  }
}
```

Run `/mcp` in Gemini CLI to check the connection.

</TabItem>
<TabItem value="other" label="Other clients">

Any MCP client that supports remote (Streamable HTTP) servers can connect with:

```text
URL:     https://app.maxun.dev/api/mcp
Header:  x-api-key: your_api_key_here
```

If your client only supports local (stdio) servers, run the server through [`mcp-remote`](https://www.npmjs.com/package/mcp-remote):

```bash
npx -y mcp-remote https://app.maxun.dev/api/mcp --header "x-api-key:your_api_key_here"
```

</TabItem>
</Tabs>

:::tip Work with a team's robots
Add an `x-team-id` header with your team ID to use a [team's](/teams-management) robots instead of your personal ones. See [Use a team's robots](/mcp/setup#use-a-teams-robots).
:::

### Try it

Ask your agent something like:

- "Scrape https://news.ycombinator.com and list the top 10 stories as a table"
- "Search the web for the latest Maxun release notes and summarize them"
- "Crawl https://docs.maxun.dev and tell me which pages mention webhooks"
- "Get the name, price and rating of every product on https://example.com/shop"
- "Run my 'Competitor prices' robot and show me what changed"

See [MCP Tools](/mcp/tools) for every tool and its options.

## Install an agent skill

Skills teach an agent how to run your existing Maxun robots with simple commands. Use them if you already build robots in the dashboard and want to run them from your agent.

| Agent | Install | Guide |
|---|---|---|
| Claude Code | Download the skill to `~/.claude/skills/maxun` | [Claude Code skill](/integrations/claude-code) |
| OpenClaw | `clawhub install maxun` | [OpenClaw skill](/integrations/openclaw) |

All skills are open source in the [maxun-skills](https://github.com/getmaxun/maxun-skills) repository.

## Give your agent the docs

When a coding agent writes code that uses the Maxun SDK, CLI or API, give it the docs so it uses the current API:

- **[llms.txt](https://docs.maxun.dev/llms.txt)**: a short, curated map of the docs.
- **[llms-full.txt](https://docs.maxun.dev/llms-full.txt)**: every docs page in one file.
- **Any page as Markdown**: add `.md` to its URL, for example [docs.maxun.dev/quickstart.md](https://docs.maxun.dev/quickstart.md).

To make this permanent, add a note to your project's `AGENTS.md`, `CLAUDE.md` or `.cursor/rules`:

```markdown
## Maxun
This project uses Maxun for web scraping and data extraction.
Before writing Maxun code, read https://docs.maxun.dev/llms.txt.
Fetch any docs page as Markdown by adding .md to its URL.
Read the API key from the MAXUN_API_KEY environment variable.
```

Or paste this prompt to have your agent add Maxun to your project:

<PromptCard title="Add Maxun to my project">{`Add Maxun (https://maxun.dev) to this project so it can scrape, crawl, search and extract web data.

1. Read https://docs.maxun.dev/llms.txt to learn how Maxun works. Fetch any page as Markdown by adding .md to its URL.
2. Pick the official SDK for this project's language: maxun-sdk for Node.js/TypeScript, or maxun for Python.
3. Install it and read the API key from the MAXUN_API_KEY environment variable. Never hard-code the key. If .env.example exists, add MAXUN_API_KEY to it.
4. Add a small example that scrapes https://maxun.dev into Markdown and prints the result.
5. Tell me where to get an API key: https://app.maxun.dev, under Settings → API Key.`}</PromptCard>

See [LLM-ready docs](/ai/llm-ready-docs) for everything the docs offer AI tools.

</page>

<page url="https://docs.maxun.dev/ai/llm-ready-docs" title="LLM-Ready Docs: llms.txt, Markdown Pages & Copy for AI">

# LLM-Ready Docs

The Maxun docs are written for people and for AI. Give them to ChatGPT, Claude, Cursor or your own agent so it answers questions and writes Maxun code from the current docs.

## llms.txt

[`https://docs.maxun.dev/llms.txt`](https://docs.maxun.dev/llms.txt) follows the [llms.txt standard](https://llmstxt.org). It is a short, curated guide to Maxun: the core concepts, which robot to use for which job, the options of every robot type, SDK, CLI, MCP and API usage, and links to every page.

Start here when an agent needs to understand Maxun quickly.

## llms-full.txt

[`https://docs.maxun.dev/llms-full.txt`](https://docs.maxun.dev/llms-full.txt) contains every docs page in one Markdown file. Each page is wrapped in a `<page url="..." title="...">` tag so the agent knows where each part came from.

Use it when you want to load all the docs into a model's context or index them for retrieval (RAG).

## Any page as Markdown

Add `.md` to the URL of any page to get its Markdown:

| Page | Markdown |
|---|---|
| `https://docs.maxun.dev/quickstart` | [`https://docs.maxun.dev/quickstart.md`](https://docs.maxun.dev/quickstart.md) |
| `https://docs.maxun.dev/mcp/setup` | [`https://docs.maxun.dev/mcp/setup.md`](https://docs.maxun.dev/mcp/setup.md) |
| `https://docs.maxun.dev/` | [`https://docs.maxun.dev/index.md`](https://docs.maxun.dev/index.md) |

Markdown has no navigation, scripts or styling, so it takes far fewer tokens than the HTML page.

## Copy page

Every page has a **Copy page** menu in the top right:

- **Copy page as Markdown**: copies the page so you can paste it into any chat.
- **View page as Markdown**: opens the raw Markdown.
- **Open in ChatGPT** / **Open in Claude**: starts a new chat that reads the page.

## Use the docs in your coding agent

Add this to your project's `AGENTS.md`, `CLAUDE.md` or `.cursor/rules` so your agent always checks the docs before writing Maxun code:

```markdown
## Maxun
This project uses Maxun for web scraping and data extraction.
Before writing Maxun code, read https://docs.maxun.dev/llms.txt.
Fetch any docs page as Markdown by adding .md to its URL.
Read the API key from the MAXUN_API_KEY environment variable.
```

To give your agent Maxun's tools as well as its docs, see [Add Maxun to your agent](/ai/add-to-agent).

</page>

<page url="https://docs.maxun.dev/mcp/setup" title="Set Up the Maxun MCP Server (Claude, Cursor)">

# MCP Server Setup

Maxun MCP Server lets you connect any MCP-compatible AI client (Claude Desktop, Cursor, Windsurf, Cline, Claude Code, etc.) to Maxun. Your AI client can then:

- **Scrape** any page into clean markdown, HTML, text, links or screenshots, and answer questions about it
- **Crawl** whole websites
- **Search** the web and scrape the results
- **Extract** structured data by describing it in plain language, with an AI agent
- **Run and monitor** the robots you already have, including Recorder Mode robots

See [MCP Tools](/mcp/tools) for every tool and its options.

There are two variants depending on how you use Maxun:

| | **Maxun Cloud** | **Self-Hosted (OSS)** |
|---|---|---|
| Setup | API key + URL only | Build step required |
| Auth | `x-api-key` header | Environment variable |
| Server URL | `https://app.maxun.dev/api/mcp` | Your local backend |

---

## Maxun Cloud

No build step needed. Connect directly to the cloud API with your Maxun API key.

### Prerequisites

- A Maxun Cloud account at [app.maxun.dev](https://app.maxun.dev)
- An MCP-compatible client (Claude Desktop, Cursor, Windsurf, Cline, etc.)

### 1. Generate your API Key

Go to your Maxun Dashboard and navigate to **Settings → API Key** to generate a key.

![Generate API Key](gen_api_key.png)|
:---:|
|Generate API Key|

![API Key Generated](api_key_success.png)|
:---:|
|API Key Generated|

### 2. Configure your MCP client

Choose the configuration format that matches your client.

#### Native HTTP

```json
{
  "mcpServers": {
    "maxun": {
      "url": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here"
      }
    }
  }
}
```

#### Via mcp-remote (stdio bridge)

If your client does not support the `url` + `headers` format natively, use [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) as a stdio-to-HTTP bridge — no separate install needed, `npx` handles it automatically.

```json
{
  "mcpServers": {
    "maxun": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://app.maxun.dev/api/mcp",
        "--header",
        "x-api-key:your_api_key_here"
      ]
    }
  }
}
```

Most clients store this config as a JSON file. For example, in **Claude Desktop** it is at:
- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`

Check your client's documentation for the exact config file location.

### 3. Restart your client

After saving the config, restart the application. On success you will see the Maxun tools available in the client.

![MCP Server Initialized](mcp_init_success.png)

### Use a team's robots

By default the MCP server works with your personal robots. To work with a [team's](/teams-management) robots instead, add an `x-team-id` header with the team ID (shown on the **Teams** page of the dashboard). Your team role applies: viewers can list and read robots and runs, members can also create and run them.

```json
{
  "mcpServers": {
    "maxun": {
      "url": "https://app.maxun.dev/api/mcp",
      "headers": {
        "x-api-key": "your_api_key_here",
        "x-team-id": "your_team_id_here"
      }
    }
  }
}
```

With `mcp-remote`, add another `"--header", "x-team-id:your_team_id_here"` pair to `args`.

### Claude Code

Add the server from your terminal:

```bash
claude mcp add --transport http maxun https://app.maxun.dev/api/mcp --header "x-api-key: your_api_key_here"
```

---

## Self-Hosted (OSS)

Use this if you are running the open-source Maxun instance locally or on your own server.

### Prerequisites

- Maxun OSS running locally (default: `http://localhost:8080`)
- Node.js installed

### 1. Build the MCP worker

Run the following command from the root of your Maxun project:

```bash
npm run mcp:build
```

This generates `dist/mcp-worker.js` in your project root.

### 2. Generate your API Key

Generate the API Key from your self-hosted Maxun Dashboard under **Settings → API Key**.

### 3. Configure your MCP client

Add the following to your client's MCP server config. For example, in Claude Desktop go to **File → Settings → Developer → Edit Config**:

```json
{
  "mcpServers": {
    "maxun": {
      "command": "node",
      "args": [
        "/path/to/your/project/dist/mcp-worker.js"
      ],
      "env": {
        "MCP_API_KEY": "your_api_key_here",
        "BACKEND_URL": "your_backend_url_here",
        "MCP_WORKER": "true"
      }
    }
  }
}
```

Replace `/path/to/your/project` with the absolute path to your Maxun project directory, and `your_backend_url_here` with your backend URL (e.g. `http://localhost:8080`).

After saving, restart your client. You'll see the Maxun tools available on successful configuration.

### 4. (Optional) Configure an LLM

Self-hosted Maxun uses your own LLM for AI features. To use `maxun_agent`, the `summary` output format or smart queries in `maxun_scrape`, add these to the `env` block above:

| Variable | Description |
|---|---|
| `MAXUN_LLM_PROVIDER` | `anthropic`, `openai` or `ollama` |
| `MAXUN_LLM_MODEL` | Model name, e.g. `claude-sonnet-4-5`, `gpt-4o` or `llama3.1` |
| `MAXUN_LLM_API_KEY` | API key for Anthropic or OpenAI |
| `MAXUN_LLM_BASE_URL` | Base URL for Ollama, e.g. `http://localhost:11434` |

```json
"env": {
  "MCP_API_KEY": "your_api_key_here",
  "BACKEND_URL": "http://localhost:8080",
  "MCP_WORKER": "true",
  "MAXUN_LLM_PROVIDER": "anthropic",
  "MAXUN_LLM_MODEL": "claude-sonnet-4-5",
  "MAXUN_LLM_API_KEY": "your_anthropic_key_here"
}
```

The key is sent to your own Maxun backend only. Everything else, including scrape, crawl and search, works without an LLM.

:::note Upgrading
After updating your Maxun project, run `npm run mcp:build` again and restart your client to get the latest tools.
:::

</page>

<page url="https://docs.maxun.dev/mcp/tools" title="Maxun MCP Tools Reference">

# MCP Tools

The Maxun MCP Server gives your AI client two kinds of tools:

- **One-shot tools** that do the work for you: scrape a page, crawl a site, search the web, or describe the data you want and let an AI agent extract it. No robot needs to exist first.
- **Robot and run tools** to list, run and inspect the robots you already have, including ones built in Recorder Mode in the dashboard.

Every tool is available on both [Maxun Cloud](https://app.maxun.dev) and self-hosted (OSS) Maxun, with the small differences noted below.

## Available Tools

| Tool | What it does |
|---|---|
| [`maxun_scrape`](#maxun_scrape) | Turn one page into markdown, HTML, text, links, a summary or screenshots, and answer questions about it |
| [`maxun_crawl`](#maxun_crawl) | Crawl a website and return the content of every page |
| [`maxun_search`](#maxun_search) | Search the web and return results, optionally scraping each one |
| [`maxun_agent`](#maxun_agent) | Describe the data you want; Maxun's AI builds an extraction robot and runs it |
| [`maxun_list_robots`](#maxun_list_robots) | List the robots in your account |
| [`maxun_get_robot`](#maxun_get_robot) | Get a robot's details |
| [`maxun_run_robot`](#maxun_run_robot) | Run a robot and return the data it extracts |
| [`maxun_get_robot_runs`](#maxun_get_robot_runs) | List a robot's runs |
| [`maxun_get_run`](#maxun_get_run) | Get a run's status and output |
| [`maxun_cancel_run`](#maxun_cancel_run) | Cancel a queued or running run |
| [`maxun_get_run_diff`](#maxun_get_run_diff) | See what changed since the previous run |
| [`maxun_get_robot_summary`](#maxun_get_robot_summary) | Run counts, success rate and data totals for a robot |

:::info Recorder Mode
Robots that record your clicks in a browser (Recorder Mode) are built in the dashboard. Once built, you can run them, and read and monitor their results, with the robot and run tools.
:::

:::note Renamed tools
All tools now start with `maxun_`. If you allow-listed tools by name in your client, update the names: `list_robots` → `maxun_list_robots`, `get_robot` → `maxun_get_robot`, `run_robot` → `maxun_run_robot`, `get_robot_runs` → `maxun_get_robot_runs`, `get_run_details` → `maxun_get_run`, `get_robot_summary` → `maxun_get_robot_summary`.
:::

## How to Use

You talk to your AI client in plain language and it picks the tool. Some examples:

| You say | Tool used |
|---|---|
| "Get me the pricing page of linear.app as markdown" | `maxun_scrape` |
| "What does the Pro plan cost on https://example.com/pricing?" | `maxun_scrape` with `smart_queries` |
| "Crawl the docs at docs.example.com, only pages under /api/" | `maxun_crawl` |
| "Find the 5 most recent articles about web scraping regulation this week" | `maxun_search` |
| "Get the name, price and rating of the first 20 laptops on bestbuy.com" | `maxun_agent` |
| "Run my Amazon price scraper" | `maxun_list_robots` → `maxun_run_robot` |
| "What changed on the last run of my competitor monitor?" | `maxun_get_robot_runs` → `maxun_get_run_diff` |

### Robots are saved for you

`maxun_scrape`, `maxun_crawl`, `maxun_search` and `maxun_agent` save the robot they build in your account. You can see it in the dashboard, run it again, schedule it, or turn on [monitoring](/monitoring) for it.

Scrape, crawl and search robots are named after their target, like `MCP Scrape: example.com/pricing [3fa2b1c9]`. Calling the tool again with the same arguments reuses that robot instead of creating a new one, so repeated calls build up one run history. Pass `robot_name` to choose the name yourself.

:::tip Plan limits
Saved robots count toward your plan's robot limit, and each run uses credits, just like robots you create in the dashboard or through the API.
:::

### Long-running work

Tools wait for the run to finish and return the data. Crawls and large extractions can take several minutes, which is longer than some clients wait for a tool. For those, pass `wait: false`: the tool returns a run ID straight away, and your AI client can call `maxun_get_run` later to fetch the result.

```
"Start a crawl of docs.example.com in the background, then check on it in a few minutes"
```

---

## One-shot Tools

### `maxun_scrape`

Scrapes a single page. Returns the page content as text blocks your AI client can read directly, plus any links, screenshots and smart-query answers.

| Parameter | Required | Description |
|---|---|---|
| `url` | Yes | The page to scrape |
| `formats` | No | Any of `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage`. Default `["markdown"]` |
| `smart_queries` | No | Questions to answer from the page, e.g. `"What is the price of the Pro plan?"` |
| `robot_name` | No | Name for the saved robot |

On self-hosted Maxun, `summary` and `smart_queries` need an [LLM configured for the MCP server](/mcp/setup#4-optional-configure-an-llm).

### `maxun_crawl`

Crawls a website from a start URL and returns every page it finds.

| Parameter | Required | Description |
|---|---|---|
| `url` | Yes | Where to start |
| `limit` | No | Maximum pages (default 50) |
| `max_depth` | No | How many links deep to go (default 3) |
| `mode` | No | `domain` (default), `subdomain` (also crawl subdomains) or `path` (only under the start URL's path) |
| `include_paths` | No | Only crawl paths matching these patterns, e.g. `["/docs/"]` |
| `exclude_paths` | No | Skip paths matching these patterns, e.g. `["/blog/"]` |
| `use_sitemap` | No | Find pages from `sitemap.xml` (default `true`) |
| `follow_links` | No | Find pages by following links (default `true`) |
| `respect_robots` | No | Obey `robots.txt` (default `true`) |
| `formats` | No | Output formats per page. Default `["markdown"]` |
| `wait` | No | Wait for the crawl to finish (default `true`). See [Long-running work](#long-running-work) |
| `robot_name` | No | Name for the saved robot |

Learn more about crawl options in [Crawl](/robot/crawl).

### `maxun_search`

Searches the web.

| Parameter | Required | Description |
|---|---|---|
| `query` | Yes | The search query |
| `mode` | No | `discover` (default) returns titles, URLs and snippets. `scrape` also scrapes each result page |
| `limit` | No | Maximum results (default 10) |
| `time_range` | No | `day`, `week`, `month` or `year` |
| `formats` | No | In `scrape` mode, output formats per result. Default `["markdown"]` |
| `robot_name` | No | Name for the saved robot |

### `maxun_agent`

Describe the data you want and Maxun's AI builds an extraction robot for it, the same as [AI Mode](/robot/extract/llm-extraction) in the dashboard. It then runs it and returns structured rows. Leave out `url` and the agent finds the right website itself.

| Parameter | Required | Description |
|---|---|---|
| `prompt` | Yes | What to extract, at least 15 characters. E.g. `"Get the name, price and rating of the first 20 laptops"` |
| `url` | No | Where to start. Leave it out to let the agent find the site |
| `robot_name` | No | Name for the saved robot |
| `run` | No | Run the robot once built (default `true`). Set `false` to only build it |
| `deep_extraction` | No | **Cloud only.** Follow each item to its detail page for more fields ([Deep Extraction](/deep-extraction), paid plans) |
| `wait` | No | **Cloud only.** When running, wait for the run to finish (default `true`) |

Building takes a minute or two. The robot is saved, so next time ask your AI client to run it with `maxun_run_robot` instead of building it again.

On self-hosted Maxun, `maxun_agent` uses the LLM you [configure for the MCP server](/mcp/setup#4-optional-configure-an-llm).

---

## Robot and Run Tools

### `maxun_list_robots`

Lists the robots in your account (on Cloud, the team's robots when you [connect with a team](/mcp/setup#use-a-teams-robots)) with their ID and name. Your AI client uses this to turn a robot's name into its ID, so you can refer to robots by name.

### `maxun_get_robot`

| Parameter | Required | Description |
|---|---|---|
| `robot_id` | Yes | The robot ID |

### `maxun_run_robot`

Runs a robot or agent and returns its data. Waits up to 30 minutes on Cloud.

| Parameter | Required | Description |
|---|---|---|
| `robot_id` | Yes | The robot ID |
| `formats` | No | Override the output formats of a scrape, crawl or search robot |
| `params` | No | **Cloud only.** Input parameters for the robot's workflow |
| `wait` | No | Wait for the run to finish (default `true`) |

### `maxun_get_robot_runs`

Lists a robot's runs with their ID, status and timestamps.

| Parameter | Required | Description |
|---|---|---|
| `robot_id` | Yes | The robot ID |
| `limit` | No | **Cloud only.** Maximum runs (default 20, max 100) |

### `maxun_get_run`

Gets a run's status, and once it has finished, all of its data.

| Parameter | Required | Description |
|---|---|---|
| `run_id` | Yes | The run ID |
| `robot_id` | Self-hosted only | The robot the run belongs to |

### `maxun_cancel_run`

Cancels a queued or running run.

| Parameter | Required | Description |
|---|---|---|
| `run_id` | Yes | The run ID |
| `robot_id` | Self-hosted only | The robot the run belongs to |

### `maxun_get_run_diff`

For robots with [monitoring](/monitoring), shows what changed between a run and the previous successful run.

| Parameter | Required | Description |
|---|---|---|
| `run_id` | Yes | The run ID |
| `robot_id` | Self-hosted only | The robot the run belongs to |
| `format` | No | Only show changes for one format, e.g. `markdown` or `captured-list` |

### `maxun_get_robot_summary`

Total, successful and failed runs, success rate, and how much data the robot has extracted.

| Parameter | Required | Description |
|---|---|---|
| `robot_id` | Yes | The robot ID |

---

## Common Workflows

### Research a topic
```
"Search for the latest news about EU AI Act enforcement this month, scrape the top 5 results and summarize them"
```
`maxun_search` (scrape mode) → your AI client summarizes the pages.

### Build a reusable extractor
1. **"Get all job titles, locations and salaries from https://example.com/careers"** — `maxun_agent` builds and runs a robot
2. **"Run the careers robot again"** — `maxun_run_robot` reuses it
3. Schedule it or turn on monitoring in the dashboard

### Watch a page for changes
1. **"Scrape https://example.com/pricing"** — `maxun_scrape` saves a robot
2. Turn on monitoring for it in the dashboard, or scrape it again later
3. **"What changed on the pricing page since last time?"** — `maxun_get_robot_runs` → `maxun_get_run_diff`

### Debug a failed run
1. **"Show me the runs for robot [name]"** — find the failed run
2. **"Get the details of run [run-id]"** — see the error
3. **"Run robot [name] again"** — retry

---

## Tips for Better Prompts

- **Give the URL when you know it.** `maxun_scrape` is the fastest way to read one page.
- **Be specific with `maxun_agent`.** Name the fields you want and how many items, e.g. "name, price, rating and URL of the first 50 products".
- **Skip robot IDs.** Refer to robots by name; your AI client will look them up.
- **Use background runs for big crawls**, then ask your client to check on them.

## Error Handling

Errors from Maxun are returned to your AI client with the reason, so it can explain them or fix its request.

| Error | Cause |
|---|---|
| `Robot not found` | The robot ID doesn't exist or belongs to another account or team |
| `API key is invalid` | Check your key in the Maxun Dashboard |
| `Permission denied` | Your team role can view robots but not run them |
| `Robot limit reached` / `Insufficient credits` | Your plan's limits were reached |
| `A robot named "…" already exists` | Pick a different `robot_name` |
| `The agent could not be built` | The agent couldn't work out how to extract that data; rephrase the prompt or give a `url` |
| `maxun_agent needs an LLM` | Self-hosted: configure an LLM for the MCP server |
| `Run failed` | The robot hit an error during extraction; check the run details |
| `Execution timeout` | The run didn't finish within 30 minutes; use `wait: false` and poll with `maxun_get_run` |

</page>

<page url="https://docs.maxun.dev/llm-prompts" title="How to Write Prompts for AI Web Data Extraction">

# Writing Effective LLM Extraction Prompts

This guide applies when using LLM-powered extraction, whether through the AI Mode or the SDK's LLM extraction feature.

## Prompts That Work

**Be Specific About Fields**
- ✅ "Extract product names, prices, and ratings"
- ❌ "Extract product data"

**Specify Quantity**
- ✅ "Extract first 20 companies"
- ✅ "Get top 50 articles"
- ❌ "Extract some items"

**Use Clear Field Names**
- ✅ "Extract company name, website URL, and description"
- ❌ "Extract company info"

**Target List Data**
- ✅ "Extract all job postings with title, company, and location"
- ✅ "Get product listings with name and price"

## Prompts to Avoid

**Multi-Step Workflows**
- ❌ "Login and then extract data"
- ❌ "Click the button and extract results"

**Multiple Page Types**
- ❌ "Extract from homepage and product pages"
- ❌ "Get data from different sections"

**Conditional Logic**
- ❌ "Extract only products cheaper than $50"
- ❌ "Get articles published this week"

**Data Transformations**
- ❌ "Calculate the average price"
- ❌ "Convert prices to EUR"

**Single Item Extraction**
- ❌ "Get the CEO name"
- ❌ "Extract the main headline"

## When to Use LLM Extraction

1. You want to quickly extract list data from a single page
2. You want to avoid manually creating extraction selectors
3. You're extracting common patterns (products, articles, listings)

## When Not to Use LLM Extraction

1. You need multi-step workflows (logins, navigation between pages)
2. You need form submissions before extraction
3. You need to extract from multiple different page types
4. You need conditional logic or data transformations

For these use cases, use Recorder Mode or the SDK's manual extraction methods to create Extract robots.

</page>

<page url="https://docs.maxun.dev/api/api" title="Get Your Maxun API Key & Authenticate">

# API Key & Authentication

Maxun lets you get the data your robot extracted and run robots via API. Every request is authenticated with your Maxun API key.

## Base URL and Authentication

| | Maxun Cloud | Self-hosted |
|---|---|---|
| API | [Cloud API](/api/cloud-api) | [Open Source API](/api/open-source-api) |
| Base URL | `https://app.maxun.dev/api/v1` | Your `BACKEND_URL` + `/api` (default `http://localhost:8080/api`) |
| Auth header | `x-api-key: <your-api-key>` or `Authorization: Bearer <your-api-key>` | `x-api-key: <your-api-key>` |

For example, to list your robots on Maxun Cloud:

```bash
curl https://app.maxun.dev/api/v1/robots \
  -H "x-api-key: YOUR_API_KEY"
```

A request without a key returns `401`, and an invalid key returns `403`.

Available endpoints:

1. [Cloud API](./cloud-api.md): robots, agents, runs, schedules, monitoring and webhooks on Maxun Cloud
2. [Open Source API](./open-source-api.md): robots, runs and schedules on self-hosted Maxun
3. [Webhooks](./webhooks.md): receive results the moment a run finishes

## 1. Generate API Key
You can find your API key in the "API Key" section on Maxun Dashboard.

![Generate API Key](gen_api_key.png)|
:---:|
|Generate API Key|

![API Key Generated](api_key_success.png)|
:---:|
|API Key Generated|

## 2. Authorize
1. Go to "API Key" tab on the dashboard's sidebar and select the option "test your API".
2. Click `Authorize`, paste the API Key you copied in step 1 and click `Authorize` to save.
![Authorize](api_auth.png)

## 3. Live Test
Click on the **Try it out** button present within each endpoint to get the output. To execute certain endpoints, you may need to provide specific parameters.

![Try it out](try-it-out.png)

## Where Else the API Key Is Used

The same key works across every Maxun interface:

- **SDK:** set `MAXUN_API_KEY` or pass `apiKey` / `api_key`. See [Node.js SDK](/sdk/node-sdk/sdk-overview) and [Python SDK](/sdk/python-sdk/sdk-overview).
- **CLI:** `maxun login --api-key <your-api-key>`. See [CLI](/cli/cli-overview).
- **MCP:** send it as the `x-api-key` header. See [MCP Setup](/mcp/setup).
- **Chrome extension:** connect it to [extract data behind login](/extract-login).

</page>

<page url="https://docs.maxun.dev/api/cloud-api" title="Cloud API: Robots, Agents, Runs, Schedules, Monitoring & Webhooks">

# Cloud API

Create robots and agents, run and schedule them, watch for changes and manage webhooks on [Maxun Cloud](https://app.maxun.dev).

:::note
From October 2026, the original `/api/robots` API is deprecated on Maxun Cloud. Use the v1 endpoints on this page.
:::

<ApiSections items={[
  { title: 'Basics', href: '#basics', description: 'Base URL, authentication, teams, errors and limits' },
  { title: 'Account', href: '#account', description: 'Your plan, credits, usage and queue',},
  { title: 'Robot', href: '#robots', description: 'Scrape, crawl, search and document robots',  },
  { title: 'Agent', href: '#agents', description: 'Web extraction agents using prompts with AI' },
  { title: 'Run', href: '#runs', description: 'Fetch data collected by robots & agents' },
  { title: 'Schedule', href: '#schedules', description: 'Extract data on a schedule', },
  { title: 'Monitor', href: '#monitoring', description: 'Detect websites changes and get alerts', },
  { title: 'Webhooks', href: '#webhooks', description: 'Get notified when runs finish', },
]} />

## Basics

### Base URL

```
https://app.maxun.dev/api/v1
```

### Authentication

Send your [API key](/api/api) as `Authorization: Bearer <api-key>` or `x-api-key: <api-key>`.

<Tabs groupId="lang" queryString>
  <TabItem value="curl" label="cURL">

```bash
curl https://app.maxun.dev/api/v1/robots \
  -H "Authorization: Bearer YOUR_API_KEY"
```

  </TabItem>
  <TabItem value="js" label="JavaScript">

```js
const res = await fetch('https://app.maxun.dev/api/v1/robots', {
  headers: { Authorization: `Bearer ${process.env.MAXUN_API_KEY}` },
});
const { data: robots } = await res.json();
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
import os, requests

res = requests.get(
    "https://app.maxun.dev/api/v1/robots",
    headers={"Authorization": f"Bearer {os.environ['MAXUN_API_KEY']}"},
)
robots = res.json()["data"]
```

  </TabItem>
</Tabs>

### Teams

Requests use your personal robots. To use a team's robots, add `x-team-id: <team-id>`. The Team ID is on the **Teams** page of the dashboard and in [`GET /me`](#get-me). Your team role applies: viewers can read, members can also create, update and run, and admins can also delete.

### Responses

Every response is JSON with a `success` flag:

```json
{ "success": true, "data": { … } }
{ "success": false, "error": "Robot not found" }
```

| Status | Meaning |
|---|---|
| `200` / `201` | Success / created |
| `202` | Accepted; the work continues in the background |
| `400` | Invalid request; `error` says what to fix |
| `401` | API key missing |
| `402` | Not enough credits |
| `403` | Invalid API key, plan limit reached, or your team role doesn't allow it |
| `404` | Not found |
| `409` | Conflict, such as a duplicate robot name or a run that has already finished |
| `429` | Rate limited; wait for the `Retry-After` seconds |

### Pagination

Lists take `limit` (default `25`, max `100`) and `offset`, newest first. A page shorter than `limit` is the last one.

### Rate limits

Limits are per minute and scale with your plan. Responses carry `X-RateLimit-Limit` and `X-RateLimit-Remaining`.

## Account

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/me` | Your account, plan and teams |
| `GET` | `/usage` | Plan, credits and robot, agent and run counts |
| `GET` | `/usage/rate-limits` | Your current rate limits |
| `GET` | `/queue` | Runs running and waiting |

### Get me

`GET /me`

```json
{
  "success": true,
  "data": {
    "email": "you@example.com",
    "plan": "GROWTH",
    "credits": 3120.5,
    "expiresAt": "2026-11-13T12:09:17.759Z",
    "teams": [{ "id": "864e4c60-…", "name": "Acme data", "role": "ADMIN", "isOwner": true }]
  }
}
```

### Get usage

`GET /usage`. With `x-team-id`, the plan and credits are the team owner's and the counts are the team's.

```json
{
  "success": true,
  "data": {
    "scope": { "type": "personal" },
    "plan": "GROWTH",
    "credits": 3120.5,
    "billingCycle": "MONTHLY",
    "expiresAt": "2026-11-13T12:09:17.759Z",
    "robots": 14,
    "agents": 3,
    "runs": 1287
  }
}
```

`robots` includes agents. `runs` counts runs to date.

### Rate limits and queue

- `GET /usage/rate-limits` returns each limit with `category`, `limitPerMinute` and `remaining`.
- `GET /queue` returns `running`, `queued`, `oldestQueuedAt` and `lastSuccessAt`. Runs wait in the queue when all your browser slots are busy.

## Robots

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/robots` | List robots |
| `GET` | `/robots/{id}` | Get a robot |
| `POST` | `/robots/scrape` | Create a scrape robot |
| `POST` | `/robots/crawl` | Create a crawl robot |
| `POST` | `/robots/search` | Create a search robot |
| `POST` | `/robots/document-extract` | Create a document extract robot |
| `POST` | `/robots/document-parse` | Create a document parse robot |
| `PATCH` | `/robots/{id}` | Update a robot |
| `POST` | `/robots/{id}/duplicate` | Copy a robot onto another URL |
| `DELETE` | `/robots/{id}` | Delete a robot and its runs |
| `PUT` | `/robots/{id}/document` | Replace a document robot's file |
| `PATCH` | `/robots/{id}/document-prompt` | Change a document extract robot's prompt |

Robots built in the recorder are created in the dashboard; every other endpoint works with them.

### The robot object

```json
{
  "id": "4db5f26d-6f37-4c21-9f9c-e06c3bd092d9",
  "name": "Pricing page",
  "type": "scrape",
  "status": "ready",
  "error": null,
  "url": "https://example.com/pricing",
  "description": null,
  "smartQueries": "What is the price of the Pro plan?",
  "formats": ["markdown", "links"],
  "teamId": null,
  "createdAt": "2026-10-14T12:09:22.797Z",
  "updatedAt": "2026-10-14T13:10:08.709Z",
  "schedule": null,
  "monitoring": { "compareRuns": true, "monitoringPrompt": null, "alertsEnabled": false, "alertEmail": null },
  "webhooks": []
}
```

| Field | Description |
|---|---|
| `type` | `extract` (built in the recorder), `agent`, `scrape`, `crawl`, `search`, `doc-extract` or `doc-parse` |
| `status` | `ready`. Agents can also be `creating` or `failed`, with the reason in `error` |
| `smartQueries` | Scrape robots only: questions answered from the page |
| `prompt`, `deepExtraction` | Agents only |
| `schedule`, `monitoring`, `webhooks` | See [Schedules](#schedules), [Monitoring](#monitoring) and [Webhooks](#webhooks) |

### List and get robots

`GET /robots` takes `type`, `q` (name search), `scheduled` (`true` or `false`), `limit` and `offset`. `GET /robots/{id}` returns one robot.

### Create a scrape robot

`POST /robots/scrape`

| Field | Required | Description |
|---|---|---|
| `url` | Yes | Page to scrape |
| `name` | No | Defaults to `Scrape Robot - <hostname>` |
| `formats` | No | Any of `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage`. Default `["markdown"]` |
| `smartQueries` | No | Questions to answer from the page |

<Tabs groupId="lang" queryString>
  <TabItem value="curl" label="cURL">

```bash
curl -X POST https://app.maxun.dev/api/v1/robots/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing", "formats": ["markdown", "links"] }'
```

  </TabItem>
  <TabItem value="js" label="JavaScript">

```js
const res = await fetch('https://app.maxun.dev/api/v1/robots/scrape', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.MAXUN_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com/pricing', formats: ['markdown', 'links'] }),
});
const { data: robot } = await res.json();
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
res = requests.post(
    "https://app.maxun.dev/api/v1/robots/scrape",
    headers={"Authorization": f"Bearer {os.environ['MAXUN_API_KEY']}"},
    json={"url": "https://example.com/pricing", "formats": ["markdown", "links"]},
)
robot = res.json()["data"]
```

  </TabItem>
</Tabs>

### Create a crawl robot

`POST /robots/crawl` with `url`, an optional `name` and `crawlConfig`. Send `"crawlConfig": {}` to use the defaults.

| `crawlConfig` option | Default | Description |
|---|---|---|
| `mode` | `"domain"` | Links to follow: `"domain"`, `"subdomain"` or `"path"` |
| `limit` | `50` | Maximum pages |
| `maxDepth` | `3` | How many links away from the start page to go |
| `includePaths` / `excludePaths` | `[]` | URL patterns to include or skip, e.g. `["/blog/*"]` |
| `useSitemap` | `true` | Also find pages through `sitemap.xml` |
| `followLinks` | `true` | Follow links found on each page |
| `respectRobots` | `true` | Obey `robots.txt` |
| `outputFormats` | | What to capture from each page, from the scrape `formats` list |

### Create a search robot

`POST /robots/search` with an optional `name` and `searchConfig`:

| `searchConfig` option | Default | Description |
|---|---|---|
| `query` | required | What to search for |
| `mode` | `"discover"` | `"discover"` returns titles, URLs and snippets; `"scrape"` also opens each result |
| `limit` | `10` | Number of results |
| `filters.timeRange` | any time | `"day"`, `"week"`, `"month"` or `"year"` |
| `outputFormats` | | In scrape mode, what to capture from each result |

### Create a document robot

Send `multipart/form-data` with a `file` (PDF, DOCX, XLSX, CSV, JPG or PNG).

- `POST /robots/document-extract` takes `prompt` (the fields to extract) and an optional `name`. It returns the robot with its `extractionSchema`.
- `POST /robots/document-parse` takes `name` and `outputFormats` (any of `markdown`, `html`, `links`, `summary`).

```bash
curl -X POST https://app.maxun.dev/api/v1/robots/document-extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -F file=@invoice.pdf \
  -F prompt="Invoice number, date, vendor and total amount"
```

`PUT /robots/{id}/document` replaces the stored file. `PATCH /robots/{id}/document-prompt` with `{ "prompt": "…" }` changes a document extract robot's fields.

### Update, duplicate and delete

- **`PATCH /robots/{id}`** takes any of `name`, `url`, `formats` and `smartQueries` (scrape), `crawlConfig` (crawl) and `searchConfig` (search). Configs are merged into the current ones.
- **`POST /robots/{id}/duplicate`** with `url` (and optionally `name`) copies a robot onto a new URL. The schedule and webhooks are not copied.
- **`DELETE /robots/{id}`** deletes the robot and its runs. Cancel queued or running runs first.

## Agents

An agent is a robot built from a prompt, like AI Mode in the dashboard.

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/agents` | Create an agent |
| `GET` | `/agents` | List agents |
| `GET` `PATCH` `DELETE` | `/agents/{id}` | Get, rename or delete an agent |
| `POST` `GET` | `/agents/{id}/runs` | Run an agent or list its runs |

Agents are robots, so the [schedule](#schedules), [monitoring](#monitoring) and [webhook](#webhooks) endpoints also work under `/agents/{id}/…`.

### Create an agent

`POST /agents`

| Field | Required | Description |
|---|---|---|
| `prompt` | Yes | What to extract, in plain English |
| `url` | No | Where to start; leave it out to let the agent find the site |
| `name` | No | Defaults to `LLM Extract: <prompt>` |
| `deepExtraction` | No | Also open each item's detail page (paid plans). See [Deep Extraction](/deep-extraction) |

<Tabs groupId="lang" queryString>
  <TabItem value="curl" label="cURL">

```bash
curl -X POST https://app.maxun.dev/api/v1/agents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "prompt": "Job title, location and team for every open role", "url": "https://example.com/careers" }'
```

  </TabItem>
  <TabItem value="js" label="JavaScript">

```js
const res = await fetch('https://app.maxun.dev/api/v1/agents', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.MAXUN_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    prompt: 'Job title, location and team for every open role',
    url: 'https://example.com/careers',
  }),
});
const { id } = await res.json();
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
res = requests.post(
    "https://app.maxun.dev/api/v1/agents",
    headers={"Authorization": f"Bearer {os.environ['MAXUN_API_KEY']}"},
    json={
        "prompt": "Job title, location and team for every open role",
        "url": "https://example.com/careers",
    },
)
agent_id = res.json()["id"]
```

  </TabItem>
</Tabs>

### Wait until it's ready

Building takes a minute or two, so `POST /agents` returns `202` with the agent in `status: "creating"`. Poll `GET /agents/{id}` until `status` is `ready`, or `failed` with the reason in `error`. Then run, schedule and monitor it like any robot. Failed agents don't count toward your plan limits.

## Runs

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/robots/{id}/runs` | Start a run |
| `GET` | `/runs/{runId}` | Get a run and its data |
| `GET` | `/runs` | List runs |
| `GET` | `/robots/{id}/runs` | List a robot's runs |
| `GET` | `/runs/active` | List queued and running runs |
| `POST` | `/runs/{runId}/cancel` | Cancel a run |
| `DELETE` | `/runs/{runId}` | Delete a run |

### Start a run

`POST /robots/{id}/runs` takes optional `formats` (to override the robot's formats) and `params` (input parameter values). It returns `202` at once. If your browser slots are busy, `status` is `queued` and the run starts when one frees up. Each run needs at least 1 credit.

<Tabs groupId="lang" queryString>
  <TabItem value="curl" label="cURL">

```bash
curl -X POST https://app.maxun.dev/api/v1/robots/ROBOT_ID/runs \
  -H "Authorization: Bearer YOUR_API_KEY"
```

  </TabItem>
  <TabItem value="js" label="JavaScript">

```js
const headers = { Authorization: `Bearer ${process.env.MAXUN_API_KEY}` };

const start = await fetch(`https://app.maxun.dev/api/v1/robots/${robotId}/runs`, { method: 'POST', headers });
const { url } = await start.json();

let run;
do {
  await new Promise((resolve) => setTimeout(resolve, 3000));
  ({ data: run } = await (await fetch(url, { headers })).json());
} while (['queued', 'running'].includes(run.status));

console.log(run.status, run.data.markdown);
```

  </TabItem>
  <TabItem value="python" label="Python">

```python
import time

headers = {"Authorization": f"Bearer {os.environ['MAXUN_API_KEY']}"}

start = requests.post(f"https://app.maxun.dev/api/v1/robots/{robot_id}/runs", headers=headers)
url = start.json()["url"]

while True:
    time.sleep(3)
    run = requests.get(url, headers=headers).json()["data"]
    if run["status"] not in ("queued", "running"):
        break

print(run["status"], run["data"]["markdown"])
```

  </TabItem>
</Tabs>

```json
{ "success": true, "id": "a30eba41-…", "status": "running", "url": "https://app.maxun.dev/api/v1/runs/a30eba41-…" }
```

### Get a run

Poll `GET /runs/{runId}` until `status` is `success`, `failed` or `aborted`, or use a [webhook](#webhooks).

```json
{
  "success": true,
  "data": {
    "id": "a30eba41-…",
    "robotId": "4db5f26d-…",
    "robotName": "Pricing page",
    "status": "success",
    "source": "api",
    "startedAt": "2026-10-14T12:10:03.361Z",
    "finishedAt": "2026-10-14T12:10:21.457Z",
    "durationMs": 18096,
    "isPartial": false,
    "hasChanges": true,
    "changedFormats": ["markdown"],
    "error": null,
    "data": {
      "textData": {},
      "listData": [],
      "crawlData": [],
      "searchData": null,
      "markdown": "# Pricing\n…",
      "html": null,
      "text": null,
      "links": ["https://example.com/signup"],
      "summary": null,
      "promptResult": "The Pro plan costs $49 per month.",
      "documentData": null
    },
    "screenshots": []
  }
}
```

| Field | Description |
|---|---|
| `source` | `api`, `sdk`, `mcp`, `cli`, `schedule` or `dashboard` |
| `isPartial` | The run stopped early but kept what it extracted |
| `data.textData`, `data.listData` | Fields and list rows captured by recorder robots and agents |
| `data.crawlData`, `data.searchData` | Crawl pages and search results |
| `data.markdown` … `data.summary` | Page content in the formats you asked for |
| `data.promptResult` | Answers to a scrape robot's `smartQueries` |
| `data.documentData` | Fields from a document extract robot |

### List, cancel and delete runs

- **`GET /runs`** and **`GET /robots/{id}/runs`** list runs without their `data`. Filter with `robotId` (on `/runs`), `status` (comma-separated), `source`, `hasChanges`, `since` and `until` (ISO 8601), plus `limit` and `offset`.
- **`GET /runs/active`** lists queued and running runs.
- **`POST /runs/{runId}/cancel`** stops a queued or running run.
- **`DELETE /runs/{runId}`** deletes a finished run.

## Schedules

| Method | Endpoint | Description |
|---|---|---|
| `PUT` | `/robots/{id}/schedule` | Set or replace a schedule |
| `GET` | `/robots/{id}/schedule` | Get a schedule, or `null` |
| `DELETE` | `/robots/{id}/schedule` | Remove a schedule |
| `GET` | `/schedules` | List scheduled robots, soonest first |

### Set a schedule

`PUT /robots/{id}/schedule`. It's the same schedule as in the dashboard.

| Field | Required | Description |
|---|---|---|
| `runEvery` | Yes | Units between runs. Max `59` for minutes and `23` for hours; `1` for weeks |
| `runEveryUnit` | Yes | `MINUTES`, `HOURS`, `DAYS`, `WEEKS` or `MONTHS` |
| `timezone` | Yes | IANA timezone, e.g. `Asia/Kolkata` |
| `atTimeStart` | No | Time to run, `HH:MM`. Default `00:00` |
| `startFrom` | No | Weekday for `WEEKS`. Default `SUNDAY` |
| `dayOfMonth` | No | Day for `MONTHS`. Default `1` |

```bash
curl -X PUT https://app.maxun.dev/api/v1/robots/ROBOT_ID/schedule \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "runEvery": 1, "runEveryUnit": "DAYS", "atTimeStart": "09:30", "timezone": "Asia/Kolkata" }'
```

The response includes the `cronExpression` and `nextRunAt`. Scheduled runs use the credits of the person who created the robot.

## Monitoring

With monitoring on, each successful run is compared with the previous one and flagged with `hasChanges`. See [Monitoring](/monitoring) for how it works.

| Method | Endpoint | Description |
|---|---|---|
| `GET` `PUT` | `/robots/{id}/monitoring` | Read or change monitoring settings |
| `GET` | `/runs/{runId}/diff` | Compare a run with the previous one |

### Change settings

`PUT /robots/{id}/monitoring`. Fields you leave out keep their value. You can also send these when creating a scrape robot, crawl robot or agent.

| Field | Description |
|---|---|
| `compareRuns` | Compare each run with the previous one |
| `monitoringPrompt` | What counts as a meaningful change, in plain English (AI-judged) |
| `alertsEnabled` | Email when a run has changes. Needs `compareRuns` |
| `alertEmail` | Where to send alerts. Defaults to the robot owner |

### Read a diff

`GET /runs/{runId}/diff` returns a line diff for each changed format (`text`, `markdown`, `html`, `captured-text`, `captured-list`). Add `?format=markdown` for one format. To find runs with changes, use `GET /runs?hasChanges=true`.

```json
{
  "success": true,
  "data": {
    "runId": "2222…",
    "previousRunId": "1111…",
    "hasChanges": true,
    "changedFormats": ["markdown"],
    "diffs": [
      {
        "format": "markdown",
        "changes": [
          { "value": "Pro: $39\n", "added": false, "removed": true },
          { "value": "Pro: $49\n", "added": true, "removed": false }
        ]
      }
    ]
  }
}
```

## Webhooks

Webhooks POST to your URL when a run completes or fails. See [Webhooks](/api/webhooks) for the payload.

| Method | Endpoint | Description |
|---|---|---|
| `GET` `POST` | `/robots/{id}/webhooks` | List or add webhooks |
| `PATCH` `DELETE` | `/robots/{id}/webhooks/{webhookId}` | Update or remove a webhook |
| `POST` | `/robots/{id}/webhooks/{webhookId}/test` | Send a test event |

### Add a webhook

`POST /robots/{id}/webhooks`. A robot can have up to 10 webhooks.

| Field | Default | Description |
|---|---|---|
| `url` | required | A public `http` or `https` URL |
| `events` | both | `run_completed`, `run_failed` or both |
| `active` | `true` | `false` pauses the webhook |
| `retryAttempts` | `3` | Attempts per event (1–10) |
| `retryDelay` | `5` | Seconds before the first retry; doubles each time |
| `timeout` | `30` | Seconds to wait for your server (1–120) |

```bash
curl -X POST https://app.maxun.dev/api/v1/robots/ROBOT_ID/webhooks \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/hooks/maxun" }'
```

`POST …/{webhookId}/test` sends a sample event and returns whether your server accepted it.

</page>

<page url="https://docs.maxun.dev/api/open-source-api" title="Open Source API: Robots, Runs & Schedules for Self-Hosted Maxun">

# Open Source API

This is the REST API of self-hosted (open source) Maxun.

## Base URL and authentication

```
<BACKEND_URL>/api
```

`BACKEND_URL` defaults to `http://localhost:8080`. Send your [API key](/api/api) in the `x-api-key` header.

```bash
curl http://localhost:8080/api/robots \
  -H "x-api-key: YOUR_API_KEY"
```

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/api/robots` | [Get all robots](#1-request-get-all-robots) |
| `GET` | `/api/robots/{id}` | [Get a robot by ID](#2-request-get-robot-by-id) |
| `POST` | `/api/robots/{id}/duplicate` | [Duplicate a robot with a new target URL](#3-request-duplicate-a-robot) |
| `POST` | `/api/robots/{id}/schedule` | [Schedule recurring runs for a robot](#4-request-schedule-recurring-runs-for-a-robot) |
| `GET` | `/api/robots/{id}/schedule` | [Get a robot's schedule](#5-request-get-a-robots-schedule) |
| `DELETE` | `/api/robots/{id}/schedule` | [Delete a robot's schedule](#6-request-delete-a-robots-schedule) |
| `GET` | `/api/robots/{id}/runs` | [Get all runs for a robot](#1-request-get-all-runs-for-a-robot) |
| `POST` | `/api/robots/{id}/runs` | [Run a robot](#2-request-run-a-robot-by-id) |
| `GET` | `/api/robots/{id}/runs/{runId}` | [Get a specific run and its data](#3-request-get-a-specific-run-by-id-for-a-robot) |

For long-running robots, use [Webhooks](/api/webhooks) to receive results instead of polling.

## Robots

### 1. Request: Get All Robots
- Request type: GET
- Route: `/api/robots`
- Description: Retrieve a list of all robots.

#### Parameters
No parameters

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | A list of robots. | application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "robots": {
    "totalCount": 2,
    "items": [
      {
        "id": "0a3acb0f-4928-483b-8610-35049efa90fd",
        "name": "100 Blog Posts",
        "createdAt": 1748765979000,
        "inputParameters": [
          {
            "type": "string",
            "name": "originUrl",
            "label": "Origin URL",
            "required": true,
            "defaultValue": "https://linktr.ee/blog/trends/"
          }
        ]
      },
      {
        "id": "9d458b00-e9dd-4799-b5f2-0166565d2af3",
        "name": "HackerNews",
        "createdAt": 1748765979219,
        "inputParameters": [
          {
            "type": "string",
            "name": "originUrl",
            "label": "Origin URL",
            "required": true,
            "defaultValue": "https://news.ycombinator.com/"
          }
        ]
      }
    ]
  }
}
```
| 500 | Error retrieving robots. | application/json |

Example Value Schema
```
{
  "statusCode": 500,
  "messageCode": "error",
  "message": "Failed to retrieve robots"
}
```

### 2. Request: Get Robot By ID
- Request type: GET
- Route: `GET /api/robots/{id}`
- Description: Retrieve a robot by its ID.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot to retrieve.


#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | Robot details.| application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "robot": {
    "id": "0a3acb0f-4928-483b-8610-35049efa90fd",
    "name": "10 post link tree",
    "createdAt": 1748765979000,
    "inputParameters": [
      {
        "type": "string",
        "name": "originUrl",
        "label": "Origin URL",
        "required": true,
        "defaultValue": "https://linktr.ee/blog/trends/"
      }
    ]
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 404 | Robot not found. | application/json |

Example Value Schema
```
{
  "statusCode": 404,
  "messageCode": "not_found",
  "message": "Recording with ID not found."
}
```

### 3. Request: Duplicate a Robot
- Request type: POST
- Route: `/api/robots/{id}/duplicate`
- Description: Duplicate an existing robot to run it on a different URL with the same structure.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot to duplicate.

#### Request Body

| Name | Description |
|---|---|
| targetUrl (required) string | The new URL to target in the duplicated robot.

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 201 | Robot duplicated successfully. | application/json |

Example Value Schema
```
{
  "statusCode": 201,
  "messageCode": "success",
  "robot": {
    "id": "e4b3c2d1-0a9b-8c7d-6e5f-4g3h2i1j0k9l",
    "name": "HackerNews (show)",
    "createdAt": 1748765980000,
    "inputParameters": [
      {
        "type": "string",
        "name": "originUrl",
        "label": "Origin URL",
        "required": true,
        "defaultValue": "https://news.ycombinator.com/show"
      }
    ]
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 404 | Robot not found. | application/json |

Example Value Schema
```
{
  "statusCode": 404,
  "messageCode": "not_found",
  "message": "Robot with ID not found."
}
```
| Code | Description | Media Type 
|---|---|---|
| 500 | Internal server error. | application/json |

### 4. Request: Schedule Recurring Runs For A Robot
- Request type: POST
- Route: `/api/robots/{id}/schedule`
- Description: Create or replace a recurring schedule for a robot. The robot will run automatically based on the provided frequency. Re-posting to this endpoint replaces any existing schedule.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot to schedule.

#### Request Body

| Name | Description |
|---|---|
| runEvery (required) integer | How often the robot runs, in units of `runEveryUnit`. |
| runEveryUnit (required) string | The schedule unit. Accepted values: `MINUTES`, `HOURS`, `DAYS`, `WEEKS`, `MONTHS`. |
| startFrom (required) string | The day to start from. Accepted values: `SUNDAY`, `MONDAY`, `TUESDAY`, `WEDNESDAY`, `THURSDAY`, `FRIDAY`, `SATURDAY`. |
| atTimeStart (required) string | Start time in `HH:MM` 24-hour format. |
| atTimeEnd (required) string | End time in `HH:MM` 24-hour format. |
| timezone (required) string | IANA timezone name, such as `UTC` or `Asia/Kolkata`. |
| dayOfMonth string | Day of the month. Used only when `runEveryUnit` is `MONTHS`. |

Example Request Body
```
{
  "runEvery": 6,
  "runEveryUnit": "HOURS",
  "startFrom": "MONDAY",
  "atTimeStart": "09:00",
  "atTimeEnd": "17:00",
  "timezone": "UTC"
}
```

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | Schedule created successfully. | application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "schedule": {
    "runEvery": 6,
    "runEveryUnit": "HOURS",
    "startFrom": "MONDAY",
    "atTimeStart": "09:00",
    "atTimeEnd": "17:00",
    "timezone": "UTC",
    "cronExpression": "0 9-17/6 * * *",
    "nextRunAt": "2025-06-02T09:00:00.000Z"
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 400 | Invalid schedule parameters. | application/json |

Example Value Schema
```
{
  "statusCode": 400,
  "messageCode": "bad_request",
  "message": "Invalid schedule parameters.",
  "field": "runEveryUnit"
}
```
| Code | Description | Media Type 
|---|---|---|
| 401 | Unauthorized access. | application/json |

Example Value Schema
```
{
  "statusCode": 401,
  "messageCode": "error",
  "message": "Unauthorized"
}
```
| Code | Description | Media Type 
|---|---|---|
| 404 | Robot not found. | application/json |

Example Value Schema
```
{
  "statusCode": 404,
  "messageCode": "not_found",
  "message": "Robot with ID not found."
}
```
| Code | Description | Media Type 
|---|---|---|
| 500 | Error scheduling robot. | application/json |

Example Value Schema
```
{
  "statusCode": 500,
  "messageCode": "error",
  "message": "Failed to schedule robot"
}
```

### 5. Request: Get A Robot's Schedule
- Request type: GET
- Route: `/api/robots/{id}/schedule`
- Description: Retrieve the current recurring schedule for a robot, or `null` if no schedule is set.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot.

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | The robot's schedule. | application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "schedule": {
    "runEvery": 6,
    "runEveryUnit": "HOURS",
    "startFrom": "MONDAY",
    "atTimeStart": "09:00",
    "atTimeEnd": "17:00",
    "timezone": "UTC",
    "cronExpression": "0 9-17/6 * * *",
    "nextRunAt": "2025-06-02T09:00:00.000Z"
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 401 | Unauthorized access. | application/json |

Example Value Schema
```
{
  "statusCode": 401,
  "messageCode": "error",
  "message": "Unauthorized"
}
```
| Code | Description | Media Type 
|---|---|---|
| 404 | Robot not found. | application/json |

Example Value Schema
```
{
  "statusCode": 404,
  "messageCode": "not_found",
  "message": "Robot with ID not found."
}
```
| Code | Description | Media Type 
|---|---|---|
| 500 | Error retrieving schedule. | application/json |

Example Value Schema
```
{
  "statusCode": 500,
  "messageCode": "error",
  "message": "Failed to retrieve schedule"
}
```

### 6. Request: Delete A Robot's Schedule
- Request type: DELETE
- Route: `/api/robots/{id}/schedule`
- Description: Cancel a robot's recurring schedule. This endpoint succeeds even if the robot has no schedule.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot.

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | Schedule deleted successfully. | application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "message": "Schedule deleted successfully"
}
```
| Code | Description | Media Type 
|---|---|---|
| 401 | Unauthorized access. | application/json |

Example Value Schema
```
{
  "statusCode": 401,
  "messageCode": "error",
  "message": "Unauthorized"
}
```
| Code | Description | Media Type 
|---|---|---|
| 404 | Robot not found. | application/json |

Example Value Schema
```
{
  "statusCode": 404,
  "messageCode": "not_found",
  "message": "Robot with ID not found."
}
```
| Code | Description | Media Type 
|---|---|---|
| 500 | Error deleting schedule. | application/json |

Example Value Schema
```
{
  "statusCode": 500,
  "messageCode": "error",
  "message": "Failed to delete schedule"
}
```

## Runs

### 1. Request: Get All Runs For A Robot
- Request type: GET
- Route: `/api/robots/{id}/runs`
- Description: Retrieve all runs associated with a specific robot.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot.

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | A list of runs for the robot.| application/json |

```
{
  "statusCode": 200,
  "messageCode": "success",
  "runs": {
    "totalCount": 2,
    "items": [
      {
        "id": "a4389317-fcfc-4006-8fc0-a31b290daf41",
        "status": "success",
        "name": "10 post link tree",
        "robotId": "0a3acb0f-4928-483b-8610-35049efa90fd",
        "startedAt": "6/1/2025, 2:27:06 pm",
        "finishedAt": "6/1/2025, 2:27:24 pm",
        "runId": "b9b4cba1-40b4-4947-a7d0-6194d277d0b7",
        "runByUserId": 8,
        "runByScheduleId": null,
        "runByAPI": null,
        "data": [
          {
            "Label 1": "Linktree’s 2023 Creator Report: : How to Navigate Attention Overload and Break Through the Noise",
            "Label 2": "27 September, 5 mins"
          },
          {
            "Label 1": "Here’s what we were linking to on Linktree in 2022",
            "Label 2": "08 December, 5 mins"
          },
          {
            "Label 1": "A beginner’s guide to the creator economy",
            "Label 2": "01 July, 5 mins"
          },
          {
            "Label 1": "A beginner’s guide to the Metaverse and steps to join",
            "Label 2": "20 June, 10 mins"
          },
          {
            "Label 1": "How to support the growing Black winemaking industry in the U.S.",
            "Label 2": "09 February, 7 mins"
          },
          {
            "Label 1": "It’s year of the tiger! Here are your 2022 horoscopes for the Lunar New Year",
            "Label 2": "01 February, 8 mins"
          },
          {
            "Label 1": "How to declutter, get organized and make this year your most productive one yet",
            "Label 2": "27 January, 7 mins"
          },
          {
            "Label 1": "12 free social media management tools for every creator",
            "Label 2": "17 January, 12 mins"
          },
          {
            "Label 1": "Our predictions for 2022",
            "Label 2": "23 December, 10 mins"
          },
          {
            "Label 1": "Female athletes are leading the future of NIL sponsorships",
            "Label 2": "06 December, 7 mins"
          }
        ],
        "screenshot": null
      }
    ]
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 500 | Error retrieving robots. | application/json |

Example Value Schema
```
{
  "statusCode": 500,
  "messageCode": "error",
  "message": "Failed to retrieve runs"
}
```

### 2. Request: Run A Robot By ID
- Request type: POST
- Route: `/api/robots/{id}/runs`
- Description: When you need to execute a robot and retrieve the data it captures, you can use this endpoint to initiate a new run. 

We now offer two methods for retrieving a run's details once it is finished:
1. **Webhooks (Recommended)**: For real-time notifications, we strongly recommend using webhooks. You can configure a webhook to receive an immediate notification as soon as your robot run is complete, along with its details. Learn more about setting up and using webhooks <a href="/api/webhooks">here</a>
2. **Long Polling**: Alternatively, you can continue to poll the GET endpoint to retrieve a run's details. While functional, this method requires you to periodically check the endpoint until the run is marked as finished.

We encourage you to leverage our webhook support for a more efficient and responsive integration.


#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot to run.

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | Robot run started successfully. | application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "run": {
    "id": "1ffbc139-6b16-4164-a61e-d4813c300a66",
    "status": "success",
    "name": "10 post link tree",
    "robotId": "0a3acb0f-4928-483b-8610-35049efa90fd",
    "startedAt": "8/1/2025, 12:04:24 am",
    "finishedAt": "8/1/2025, 12:04:40 am",
    "runId": "748a9995-9749-4bf1-a235-88dc61c2dd85",
    "runByUserId": null,
    "runByScheduleId": null,
    "runByAPI": true,
    "data": [
      {
        "Label 1": "Linktree’s 2023 Creator Report: : How to Navigate Attention Overload and Break Through the Noise",
        "Label 2": "27 September, 5 mins"
      },
      {
        "Label 1": "Here’s what we were linking to on Linktree in 2022",
        "Label 2": "08 December, 5 mins"
      },
      {
        "Label 1": "A beginner’s guide to the creator economy",
        "Label 2": "01 July, 5 mins"
      },
      {
        "Label 1": "A beginner’s guide to the Metaverse and steps to join",
        "Label 2": "20 June, 10 mins"
      },
      {
        "Label 1": "How to support the growing Black winemaking industry in the U.S.",
        "Label 2": "09 February, 7 mins"
      },
      {
        "Label 1": "It’s year of the tiger! Here are your 2022 horoscopes for the Lunar New Year",
        "Label 2": "01 February, 8 mins"
      },
      {
        "Label 1": "How to declutter, get organized and make this year your most productive one yet",
        "Label 2": "27 January, 7 mins"
      },
      {
        "Label 1": "12 free social media management tools for every creator",
        "Label 2": "17 January, 12 mins"
      },
      {
        "Label 1": "Our predictions for 2022",
        "Label 2": "23 December, 10 mins"
      },
      {
        "Label 1": "Female athletes are leading the future of NIL sponsorships",
        "Label 2": "06 December, 7 mins"
      }
    ],
    "screenshot": null
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 401 | Unauthorized access. | application/json |

Example Value Schema
```
{
  "ok": false,
  "error": "Unauthorized"
}
```
| Code | Description | Media Type 
|---|---|---|
| 500 | Error running robot.| application/json |

Example Value Schema
```
{
  "statusCode": 500,
  "messageCode": "error",
  "message": "Failed to run robot"
}
```

### 3. Request: Get A Specific Run By ID For A Robot
- Request type: GET
- Route: `/api/robots/{id}/runs/{runId}`
- Description: Retrieve details of a specific run by its ID.

#### Parameters

| Name | Description |
|---|---|
| id (required) string | The ID of the robot.
| runId (required) string | The ID of the run.

#### Responses

| Code | Description | Media Type 
|---|---|---|
| 200 | Run details.| application/json |

Example Value Schema
```
{
  "statusCode": 200,
  "messageCode": "success",
  "run": {
    "id": "1ffbc139-6b16-4164-a61e-d4813c300a66",
    "status": "success",
    "name": "10 post link tree",
    "robotId": "0a3acb0f-4928-483b-8610-35049efa90fd",
    "startedAt": "8/1/2025, 12:04:24 am",
    "finishedAt": "8/1/2025, 12:04:40 am",
    "runId": "748a9995-9749-4bf1-a235-88dc61c2dd85",
    "runByUserId": null,
    "runByScheduleId": null,
    "runByAPI": true,
    "data": [
      {
        "Label 1": "Linktree’s 2023 Creator Report: : How to Navigate Attention Overload and Break Through the Noise",
        "Label 2": "27 September, 5 mins"
      },
      {
        "Label 1": "Here’s what we were linking to on Linktree in 2022",
        "Label 2": "08 December, 5 mins"
      },
      {
        "Label 1": "A beginner’s guide to the creator economy",
        "Label 2": "01 July, 5 mins"
      },
      {
        "Label 1": "A beginner’s guide to the Metaverse and steps to join",
        "Label 2": "20 June, 10 mins"
      },
      {
        "Label 1": "How to support the growing Black winemaking industry in the U.S.",
        "Label 2": "09 February, 7 mins"
      },
      {
        "Label 1": "It’s year of the tiger! Here are your 2022 horoscopes for the Lunar New Year",
        "Label 2": "01 February, 8 mins"
      },
      {
        "Label 1": "How to declutter, get organized and make this year your most productive one yet",
        "Label 2": "27 January, 7 mins"
      },
      {
        "Label 1": "12 free social media management tools for every creator",
        "Label 2": "17 January, 12 mins"
      },
      {
        "Label 1": "Our predictions for 2022",
        "Label 2": "23 December, 10 mins"
      },
      {
        "Label 1": "Female athletes are leading the future of NIL sponsorships",
        "Label 2": "06 December, 7 mins"
      }
    ],
    "screenshot": null
  }
}
```
| Code | Description | Media Type 
|---|---|---|
| 404 | Run not found. | application/json |

Example Value Schema
```
{
  "statusCode": 404,
  "messageCode": "not_found",
  "message": "Run with id not found."
}
```

</page>

<page url="https://docs.maxun.dev/api/webhooks" title="Webhooks: Get Robot Run Results in Real Time">

# Webhooks

Webhooks send a robot's results to your server the moment a run finishes, so you don't need to poll the API. Maxun sends a `POST` request with a JSON payload to the URL you configure, for successful runs and for failed runs.

You can find the webhook integration option in the integration settings for each robot.

![Maxun Webhook Integration](webhook_integrate.png)

![Maxun Webhook Configuration](webhook_config.png)

### 1. Event: Run finished
- Trigger: When a robot run finishes successfully. 

#### Payload Schema

| Field             | Type   | Description                                        |
|------------------|--------|----------------------------------------------------|
| event_type     | string | Type of event            |
| timestamp      | string | ISO timestamp when the webhook was triggered       |
| webhook_id     | string | Unique ID for this webhook                         |
| robot_id  | string | Unique identifier for the robot                    |
| run_id    | string | Unique identifier for the run                      |
| robot_name| string | Name of the robot                                  |
| status    | string | "success" or "failed"                          |
| started_at| string | ISO timestamp when the run started                 |
| finished_at| string| ISO timestamp when the run ended                   |
| extracted_data | object | Present only when run is successful         |
| error     | object | Present only when run fails                        |
| metadata  | object | Metadata like browser ID and user ID               |

---

#### Example: Successful Run

```
{
  "event_type": "run_completed",
  "timestamp": "2025-05-29T15:53:15.595Z",
  "webhook_id": "1b480cbd-c43b-4541-a69e-92d82000d992",
  "data": {
    "robot_id": "d453def3-800a-4b6c-ba31-854cf8dd8d26",
    "run_id": "110c4dae-c39b-4b30-a932-eff1022e4bb0",
    "robot_name": "Test Robot",
    "status": "test",
    "started_at": "2025-05-29T15:52:30.595Z",
    "finished_at": "2025-05-29T15:53:15.595Z",
    "extracted_data": {
      "captured_texts": [
        {
          "Product Name": "MacBook Pro 16-inch M3 Max",
          "Price": "$3,999.00",
          "Rating": "4.8/5 stars",
        }
      ],
      "captured_lists": {
        "list_1": [
          {
            "Rank": "1",
            "Product": "MacBook Air M2",
            "Category": "Laptops",
            "Units Sold": "2,847",
            "Revenue": "$2,847,000"
          }
        ],
        "list_0": [
          {
            "Customer": "Sarah M.",
            "Rating": "5 stars",
            "Review": "Absolutely love my new MacBook! The battery life is incredible and the performance is outstanding.",
            "Date": "2024-12-15",
            "Verified Purchase": "Yes"
          }
        ]
      },
      "total_rows": 5,
      "captured_texts_count": 3,
      "captured_lists_count": 2,
      "screenshots_count": 5
    },
    "metadata": {
      "browser_id": "d27ace57-75cb-441c-8589-8ba34e52f7d1",
      "user_id": 108
    }
  }
}
```

### 2. Event: Run failed
- Trigger: When a robot run fails. 

#### Payload Schema

| Field        | Type     | Description                                            |
| ------------ | -------- | ------------------------------------------------------ |
| event_type     | string | Type of event            |
| timestamp      | string | ISO timestamp when the webhook was triggered       |
| webhook_id     | string | Unique ID for this webhook                         |
| robotId    | string        | Unique identifier for the robot                        |
| runId      | string        | Unique identifier for the run                          |
| robotName  | string        | Name of the robot                                      |
| status     | string        | Status of the run                  |
| message | string        | Description of the error                               |
| type    | string        | Type/category of error                                 |
| stack   | string      | Stack trace of the error                               |
| metadata   | object      | Optional metadata like browserId, userId, etc.     |


#### Example: Failed Run

```
{
  "event_type": "run_failed",
  "timestamp": "2025-05-29T15:53:15.595Z",
  "webhook_id": "1b480cbd-c43b-4541-a69e-92d82000d992",
  "data": {
    "robotId": "d453def3-800a-4b6c-ba31-854cf8dd8d26",
    "runId": "110c4dae-c39b-4b30-a932-eff1022e4bb0",
    "robotName": "Test Robot",
    "status": "failed",
    "error": {
      "message": "Timeout exceeded while waiting for element",
      "type": "TimeoutError",
      "stack": "TimeoutError: ...\n    at async Page.waitForSelector..."
    },
    "metadata": {
      "browserId": "d27ace57-75cb-441c-8589-8ba34e52f7d1",
      "userId": 108
    }
  }
}
```

</page>

<page url="https://docs.maxun.dev/sdks" title="Maxun SDKs & CLI: Node.js, Python and Terminal">

# SDKs & CLI

Use Maxun from your own code or terminal. The SDKs and CLI create **robots** (saved, reusable jobs) on your Maxun account, so anything you create in code also shows up in the dashboard, and the other way round.

| | Install | Guide |
|---|---|---|
| **Node.js / TypeScript** | `npm install maxun-sdk` | [Node SDK](/sdk/node-sdk/sdk-overview) |
| **Python** | `pip install maxun` | [Python SDK](/sdk/python-sdk/sdk-overview) |
| **CLI** | `npm install -g maxun-cli` | [CLI](/cli/cli-overview) |

All of them work with Maxun Cloud and self-hosted Maxun. Get an API key from the dashboard under **Settings → API Key** (see [API Key](/api/api)).

## Quick example

<Tabs groupId="sdk-language">
<TabItem value="node" label="Node.js" default>

```javascript

const maxun = new Maxun({ apiKey: process.env.MAXUN_API_KEY });

const robot = await maxun.scrape('Maxun home', 'https://maxun.dev', { formats: ['markdown'] });
const result = await robot.run();

console.log(result.markdown);
```

</TabItem>
<TabItem value="python" label="Python">

```python
import asyncio
import os
from maxun import Maxun

async def main():
    async with Maxun(api_key=os.environ["MAXUN_API_KEY"]) as maxun:
        robot = await maxun.scrape("Maxun home", "https://maxun.dev", formats=["markdown"])
        result = await robot.run()
        print(result.markdown)

asyncio.run(main())
```

</TabItem>
<TabItem value="cli" label="CLI">

```bash
maxun login --api-key <your-api-key>
maxun --help
```

See [CLI Robots](/cli/cli-robots) to create robots from the terminal.

</TabItem>
</Tabs>

## What you can build

| Method | What the robot does |
|---|---|
| `scrape` | Turns a page into Markdown, HTML, text, links, a summary or screenshots |
| `extract` | Extracts structured data, described in plain English or with selectors |
| `crawl` | Visits many pages of a website |
| `search` | Searches the web and optionally scrapes the results |
| `documents.extract` / `documents.parse` | Extracts data from or converts a PDF, DOCX, XLSX, CSV, JPG or PNG |

Prefer plain HTTP? See the [API Reference](/api/api). Using an AI agent? See [Build with AI](/ai).

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-overview" title="Node.js Web Scraping SDK: Install & Setup">

# Maxun Node.js SDK

The Maxun Node.js SDK turns websites and documents into structured data from your JavaScript or TypeScript code. You create **robots** (saved, reusable jobs) and run them whenever you need fresh data.

```javascript

const maxun = new Maxun({ apiKey: 'your-api-key' });

const robot = await maxun.scrape('Maxun home', 'https://maxun.dev', { formats: ['markdown'] });
const result = await robot.run();

console.log(result.markdown);
```

## Installation

```bash
npm install maxun-sdk
```

## Requirements

- Node.js 18 or later
- A Maxun Cloud account or a self-hosted Maxun instance
- An API key from the [Maxun Dashboard](/api/api)

## Configuration

Pass your API key directly:

```javascript

const maxun = new Maxun({ apiKey: 'your-api-key' });
```

Or set it in the environment and create `new Maxun()` with no arguments:

```bash
MAXUN_API_KEY=your-api-key
MAXUN_TEAM_ID=your-team-uuid                      # optional, Maxun Cloud teams
MAXUN_BASE_URL=http://localhost:8080/api/sdk/     # only for self-hosted Maxun
```

```javascript
import 'dotenv/config';   // only needed if your variables are in a .env file

const maxun = new Maxun();
```

The SDK connects to Maxun Cloud by default. For a self-hosted instance, set `MAXUN_BASE_URL` or pass `baseUrl`:

```javascript
const maxun = new Maxun({ apiKey: 'your-api-key', baseUrl: 'http://localhost:8080/api/sdk/' });
```

## Everything starts from `maxun`

Each call takes the **robot name** first, then **what to work on** (a URL, a search query or a file), then an optional **options object**. It returns a [`Robot`](./sdk-robot) saved on your account.

| Call | What the robot does | Read the result from |
|---|---|---|
| [`maxun.scrape(name, url)`](./sdk-scrape) | Turns a page into Markdown, HTML, text, links, a summary or screenshots | `result.markdown`, `result.html`, ... |
| [`maxun.extract(name, url, { prompt })`](./sdk-extract) | Extracts structured data, described in plain English or with selectors | `result.listData`, `result.textData` |
| [`maxun.crawl(name, url)`](./sdk-crawl) | Visits many pages of a website | `result.crawlData` |
| [`maxun.search(name, query)`](./sdk-search) | Searches the web and optionally scrapes the results | `result.searchData` |
| [`maxun.documents.extract(name, file, prompt)`](./sdk-document) | Extracts data from a PDF, DOCX, XLSX, CSV, JPG or PNG | `result.documentData` |
| [`maxun.documents.parse(name, file)`](./sdk-document) | Converts a document to Markdown, HTML, links or a summary | `result.markdown`, ... |
| [`maxun.robots`](./sdk-robot) | Lists, finds and deletes robots of any type | |

The name is required. It is how the robot appears in the Maxun dashboard. Unknown options throw an error instead of being silently ignored.

:::tip
The examples in these docs use top-level `await`, which works in ES modules (`.mjs` files, or `"type": "module"` in `package.json`). In CommonJS, wrap the code in an `async` function.
:::

## TypeScript

The SDK is written in TypeScript and ships its own types. Options, results and errors are all typed:

```typescript

const maxun = new Maxun();
const robot: Robot = await maxun.scrape('Example', 'https://example.com');
const result: RunResult = await robot.run();
```

## Errors

Every API error is a `MaxunError` with `statusCode` and `details`. More specific errors:

| Error | When |
|---|---|
| `AuthenticationError` | The API key is missing or invalid |
| `NotFoundError` | The robot or run does not exist |
| `ConflictError` | A robot with that name already exists with different settings |
| `ValidationError` | Maxun rejected the input |
| `RunFailedError` | A run failed or was aborted |

```javascript

let robot;
try {
  robot = await maxun.scrape('Pricing page', 'https://example.com/pricing');
} catch (error) {
  if (error instanceof ConflictError) {
    robot = await maxun.robots.find('Pricing page');
  } else if (error instanceof MaxunError) {
    console.error(error.statusCode, error.message);
  } else {
    throw error;
  }
}
```

## What's next

- [Scrape](./sdk-scrape): turn pages into clean content
- [Extract](./sdk-extract): pull structured data out of pages
- [Crawl](./sdk-crawl): collect content from a whole website
- [Search](./sdk-search): search the web
- [Document](./sdk-document): extract data from and convert documents
- [Robot Management](./sdk-robot): run, schedule and manage robots
- [Monitoring](./sdk-monitoring): get notified when a page changes

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-scrape" title="Node.js: Scrape Websites to Markdown & HTML">

# Scrape

A scrape robot turns a web page into clean Markdown, HTML, plain text, a list of links, an AI summary or screenshots.

```javascript

const maxun = new Maxun();

const robot = await maxun.scrape('Example page', 'https://example.com');
const result = await robot.run();

console.log(result.markdown);
```

`maxun.scrape(name, url)` saves the robot on your account. Run it again any time with `await robot.run()`.

## Output formats

Choose what you want back with `formats`. The default is `['markdown']`.

| Format | Read it from | What you get |
|---|---|---|
| `markdown` | `result.markdown` | The page as clean Markdown, ready for an LLM |
| `html` | `result.html` | The page's HTML |
| `text` | `result.text` | The page's visible text |
| `links` | `result.links` | Every link on the page |
| `summary` | `result.summary` | An AI-written summary of the page |
| `screenshot-visible` | `result.screenshots` | A screenshot of the visible part of the page |
| `screenshot-fullpage` | `result.screenshots` | A screenshot of the whole page |

Ask for as many formats as you need in one robot:

```javascript
const robot = await maxun.scrape('Pricing page', 'https://example.com/pricing', {
  formats: ['markdown', 'links', 'screenshot-fullpage'],
});

const result = await robot.run();

console.log(result.markdown);
console.log(result.links);
console.log(result.screenshots);
```

The result only contains the formats the robot produced:

```javascript
console.log(result);
// {
//   runId: 'b05cea93-…',
//   status: 'success',
//   markdown: '# Pricing\n\n…',
//   links: [ … ],
//   screenshots: [ … ]
// }
```

### Summary

`summary` adds a short AI-written summary next to the other formats:

```javascript
const robot = await maxun.scrape('Blog post', 'https://blog.example.com/post', {
  formats: ['markdown', 'summary'],
});

const result = await robot.run();
console.log(result.summary);
```

## Smart Queries

A Smart Query is a question about the page. Maxun scrapes the page, then an LLM answers your question on every run.

```javascript
const robot = await maxun.scrape('Pricing plans', 'https://example.com/pricing', {
  smartQueries: 'List every plan with its monthly price.',
});

const result = await robot.run();
console.log(result.smartQueryResult);
```

You can also ask a different question for a single run, without changing the robot:

```javascript
const result = await robot.run({ smartQueries: 'Which plan includes SSO?' });
console.log(result.smartQueryResult);
```

:::note
`summary` and Smart Queries use an LLM. On Maxun Cloud this is handled for you. On self-hosted Maxun, add the [LLM settings](#llm-settings-self-hosted) below.
:::

## Options

| Option | Default | Description |
|---|---|---|
| `formats` | `['markdown']` | Output formats (see above) |
| `smartQueries` | none | A question an LLM answers about the page on every run |
| `monitor` | `false` | Compare every run with the previous one. See [Monitoring](./sdk-monitoring) |
| `llmProvider`, `llmModel`, `llmApiKey`, `llmBaseUrl` | none | Self-hosted Maxun only, see below |

## Running a scrape robot

```javascript
const result = await robot.run();
```

`run()` waits until the page is scraped and returns the result. You can change the formats for a single run:

```javascript
const result = await robot.run({ formats: ['html'] });
console.log(result.html);
```

If the run fails, `run()` throws `RunFailedError`. See [Robot Management](./sdk-robot) for run history, schedules and webhooks.

## LLM settings (self-hosted)

Self-hosted Maxun has no built-in LLM, so `summary` and Smart Queries need one:

```javascript
const robot = await maxun.scrape('Pricing plans', 'https://example.com/pricing', {
  smartQueries: 'List every plan with its monthly price.',
  llmProvider: 'anthropic',        // 'anthropic', 'openai' or 'ollama'
  llmApiKey: 'your-llm-api-key',   // required for anthropic and openai
  llmModel: 'claude-sonnet-4-5',   // optional
});
```

:::caution
Do not pass `llm*` options on Maxun Cloud. Cloud manages the model for you and rejects them.
:::

## Examples

### Feed a RAG pipeline

```javascript
const robot = await maxun.scrape('Docs guide', 'https://docs.example.com/guide');
const result = await robot.run();

await createEmbeddings(result.markdown);
```

### Scrape several pages

Give each robot its own name:

```javascript
const pages = {
  'Post: Launch week': 'https://blog.example.com/launch-week',
  'Post: Pricing update': 'https://blog.example.com/pricing-update',
};

for (const [name, url] of Object.entries(pages)) {
  const robot = await maxun.scrape(name, url);
  const result = await robot.run();
  await saveToDatabase(url, result.markdown);
}
```

### Use an existing robot

```javascript
const robot = await maxun.robots.find('Pricing plans');   // by name
const result = await robot.run();
```

List your scrape robots with `await maxun.scrape.list()`. See [Robot Management](./sdk-robot) for everything else you can do with a robot.

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-extract" title="Node.js: Extract Structured Data From Websites">

# Extract

An extract robot pulls structured data (product lists, prices, job postings, tables) out of web pages. There are two ways to build one with Maxun SDK:

- **With a prompt**: describe the data in plain English and Maxun builds the robot for you.
- **With selectors**: list the steps and CSS or XPath selectors yourself, for precise and predictable results.

Both return a [`Robot`](./sdk-robot) that you can run as often as you like.

## Extract with a prompt

```javascript
const robot = await maxun.extract('YC companies', 'https://www.ycombinator.com/companies', {
  prompt: 'Company name, description and batch for the first 15 companies',
});

const result = await robot.run();

for (const company of result.listData) {
  console.log(company);
}
```

```javascript
{ 'Company name': 'Airbnb', Description: 'Book accommodations around the world.', Batch: 'W09' }
{ 'Company name': 'Stripe', Description: 'Economic infrastructure for the internet.', Batch: 'S09' }
...
```

### Without a URL

On Maxun Cloud, leave the URL out and the agent searches the web for a suitable page first. Self-hosted Maxun requires a URL: [AI Mode](/robot/extract/llm-extraction) reads only the page you give it (see [Agent vs AI Mode](/robot/agent-vs-ai-mode)).

```javascript
const robot = await maxun.extract('Top AI startups', {
  prompt: 'Names and funding of the top 10 AI startups from YCombinator',
});
```

### LLM settings (self-hosted)

On Maxun Cloud, a prompt is all you need. Self-hosted Maxun has no built-in LLM, so pass one:

```javascript
const robot = await maxun.extract('Products', 'https://shop.example.com', {
  prompt: 'Product names and prices',
  llmProvider: 'anthropic',        // 'anthropic', 'openai' or 'ollama'
  llmApiKey: 'your-llm-api-key',   // required for anthropic and openai
  llmModel: 'claude-sonnet-4-5',   // optional
  // llmBaseUrl: 'http://localhost:11434',   // optional, e.g. your Ollama server
});
```

:::caution
Do not pass `llm*` options on Maxun Cloud. Cloud manages the model for you and rejects them.
:::

## Extract with selectors

`maxun.extract(name, url)` starts a robot on that page. Chain the steps you want, then finish with `.build()`:

```javascript
const robot = await maxun
  .extract('Bookstore', 'https://books.toscrape.com')
  .captureText({ Heading: 'h1' })
  .captureList({ selector: 'article.product_pod', maxItems: 20 })
  .build();

const result = await robot.run();

console.log(result.textData);   // { Heading: 'All products' }
console.log(result.listData);   // [ {...}, {...}, ... ]
```

Selector robots don't use an LLM, so they work the same on Maxun Cloud and self-hosted Maxun.

### Capture text

`captureText` reads single values. Give each field a name and a selector:

```javascript
.captureText({
  Title: 'h1.article-title',
  Author: '.author-name',
  Published: 'time',
})
```

The values are in `result.textData`. CSS and XPath selectors both work.

### Capture a list

`captureList` reads every element that matches a selector. The fields inside each item are detected automatically:

```javascript
.captureList({ selector: 'article.product_pod', maxItems: 50 })
```

The items are in `result.listData`. `maxItems` defaults to 100.

### Pagination

Maxun detects pagination automatically. To control it yourself, add `pagination`:

```javascript
// Click a "Next" button
.captureList({
  selector: 'article.product_pod',
  maxItems: 100,
  pagination: { type: 'clickNext', selector: 'li.next a' },
})

// Click a "Load more" button
.captureList({ selector: '.card', pagination: { type: 'clickLoadMore', selector: 'button.load-more' } })

// Infinite scroll
.captureList({ selector: '.feed-item', pagination: { type: 'scrollDown' } })

// First page only
.captureList({ selector: '.result', pagination: { type: 'none' } })
```

| `type` | Use it for | Needs `selector` |
|---|---|---|
| `clickNext` | A "Next" button or link | Yes |
| `clickLoadMore` | A "Load more" button | Yes |
| `scrollDown` | Infinite scroll | No |
| `scrollUp` | Content that loads when scrolling up | No |
| `none` | Reading only the first page | No |

### Browser actions

Add steps in the order they should happen:

| Step | What it does |
|---|---|
| `.navigate(url)` | Go to another page |
| `.click(selector)` | Click an element |
| `.type(selector, text)` | Type into an input. The text is stored encrypted. |
| `.waitFor(selector, 30000)` | Wait for an element to appear (milliseconds) |
| `.wait(1000)` | Pause (milliseconds) |
| `.scroll(2)` | Scroll down by a number of screen heights |
| `.captureScreenshot('name', { fullPage: true })` | Take a screenshot, returned in `result.screenshots` |

Give any capture a label by passing a name as the second argument, for example `.captureList({ selector: '.product' }, 'Products')`.

## Examples

### A list across several pages

```javascript
const robot = await maxun
  .extract('Quotes', 'https://quotes.toscrape.com')
  .captureList({
    selector: 'div.quote',
    maxItems: 50,
    pagination: { type: 'clickNext', selector: 'li.next a' },
  })
  .build();

const result = await robot.run();
console.log(`${result.listData.length} quotes`);
```

### Several pages in one robot

```javascript
const robot = await maxun
  .extract('Store overview', 'https://shop.example.com')
  .captureText({ 'Store name': 'h1' })
  .navigate('https://shop.example.com/products')
  .captureList({ selector: '.product' }, 'Products')
  .navigate('https://shop.example.com/reviews')
  .captureList({ selector: '.review' }, 'Reviews')
  .build();
```

### Log in, then extract

```javascript
const robot = await maxun
  .extract('Dashboard data', 'https://app.example.com/login')
  .type('#email', 'you@example.com')
  .type('#password', 'your-password')
  .click('button[type=submit]')
  .waitFor('.dashboard')
  .captureText({ Balance: '.balance', Plan: '.plan-name' })
  .captureScreenshot('Dashboard')
  .build();
```

### Watch for changes

Turn on [monitoring](./sdk-monitoring) to compare every run with the previous one:

```javascript
const robot = await maxun.extract('Product prices', 'https://shop.example.com/product/42', {
  prompt: 'Product name, price and availability',
  monitor: true,
});
```

For selector robots, pass `{ monitor: true }` before building:

```javascript
const robot = await maxun
  .extract('Product prices', 'https://shop.example.com', { monitor: true })
  .captureList({ selector: '.product' })
  .build();
```

## Managing extract robots

```javascript
const robots = await maxun.extract.list();            // all extract robots
const robot = await maxun.robots.find('Bookstore');   // one robot, by name
await robot.delete();
```

:::note
Building a selector robot with a name and URL that already exist returns the existing robot unchanged, even if your steps are different. The SDK warns you when this happens. Use a new name, or delete the old robot first.
:::

See [Robot Management](./sdk-robot) to run, schedule and manage robots.

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-crawl" title="Node.js Web Crawler: Crawl Entire Websites">

# Crawl

A crawl robot starts on one page, follows the links it finds and collects the content of every page it visits. Use it to grab a whole documentation site, a blog or a product catalogue.

```javascript
const robot = await maxun.crawl('Example docs', 'https://docs.example.com', { limit: 20 });

const result = await robot.run();

for (const page of result.crawlData) {
  console.log(page.metadata.url);
  console.log(page.markdown?.slice(0, 200));
}
```

## Options

Every option has a sensible default, so a name and a URL are enough to start.

| Option | Default | Description |
|---|---|---|
| `mode` | `'domain'` | Which links to follow: `'domain'`, `'subdomain'` or `'path'` (see below) |
| `limit` | `50` | Maximum number of pages to crawl |
| `maxDepth` | `3` | How many links away from the start page to go |
| `includePaths` | none | Only crawl URLs matching these patterns, e.g. `['/blog/*']` |
| `excludePaths` | none | Skip URLs matching these patterns, e.g. `['/admin/*']` |
| `useSitemap` | `true` | Also find pages through the site's `sitemap.xml` |
| `followLinks` | `true` | Follow links found on each page |
| `respectRobots` | `true` | Obey the site's `robots.txt` |
| `formats` | `['markdown']` | What to capture from each page: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` |
| `monitor` | `false` | Compare every run with the previous one. See [Monitoring](./sdk-monitoring) |

```javascript
const robot = await maxun.crawl('Docs guides', 'https://docs.example.com', {
  limit: 100,
  maxDepth: 4,
  includePaths: ['/guides/*'],
  excludePaths: ['/guides/archive/*'],
  formats: ['markdown', 'links'],
});
```

## Crawl modes

`mode` decides how far the crawl may wander from the start URL.

| Mode | Starting at `https://example.com/blog` it crawls |
|---|---|
| `domain` | Any page on `example.com` |
| `subdomain` | Pages on `example.com` and its subdomains, such as `docs.example.com` |
| `path` | Only pages under `example.com/blog` |

```javascript
const robot = await maxun.crawl('Blog', 'https://example.com/blog', { mode: 'path' });
```

## Reading the results

`result.crawlData` is an array with one entry per page:

```javascript
const result = await robot.run();

for (const page of result.crawlData) {
  console.log(page.metadata.url);
  console.log(page.metadata.title);
  console.log(page.markdown);
}
```

Each page contains the formats you asked for (`markdown`, `html`, `text`, `links`, ...) plus `metadata` with the page's `url`, `title` and other meta tags. A page that could not be loaded has an `error` instead.

:::tip
Large crawls can take a while. `run()` waits until the crawl finishes. Pass `{ timeout: 600000 }` (milliseconds) to stop waiting sooner; the crawl keeps going on Maxun and you can read it later with `await robot.getLatestRun()`.
:::

## Examples

### A whole documentation site

```javascript
const robot = await maxun.crawl('Product docs', 'https://docs.example.com', {
  limit: 200,
  maxDepth: 5,
});

const result = await robot.run();

const docs = Object.fromEntries(
  result.crawlData
    .filter((page) => page.markdown)
    .map((page) => [page.metadata.url, page.markdown]),
);
```

### Only blog posts

```javascript
const robot = await maxun.crawl('Blog posts', 'https://example.com', {
  includePaths: ['/blog/*'],
  excludePaths: ['/blog/tag/*', '/blog/page/*'],
  limit: 50,
});
```

### Keep a knowledge base up to date

Crawl once a week and see which pages were added, removed or changed:

```javascript
const robot = await maxun.crawl('Help center', 'https://help.example.com', { limit: 100, monitor: true });
await robot.schedule({ runEvery: 1, runEveryUnit: 'WEEKS', startFrom: 'MONDAY', atTimeStart: '06:00' });
```

See [Monitoring](./sdk-monitoring) for how to read the changes.

### Summarize every page

```javascript
const robot = await maxun.crawl('Docs summaries', 'https://docs.example.com', {
  formats: ['summary'],
  limit: 20,
});

const result = await robot.run();
for (const page of result.crawlData) {
  console.log(page.metadata.url, '→', page.summary);
}
```

`summary` uses an LLM. On self-hosted Maxun, also pass `llmProvider`, `llmApiKey` and optionally `llmModel` and `llmBaseUrl`. On Maxun Cloud, leave them out.

## Managing crawl robots

```javascript
const robots = await maxun.crawl.list();             // all crawl robots
const robot = await maxun.robots.find('Blog posts');
await robot.setListLimit(500);                       // crawl more pages from now on
await robot.delete();
```

See [Robot Management](./sdk-robot) to run, schedule and manage robots.

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-search" title="Node.js: Web Search & Scrape Results">

# Search

A search robot searches the web (with DuckDuckGo) and returns the results. It can also open each result and scrape its content, so you get search plus page content in one step.

```javascript
const robot = await maxun.search('AI news', 'AI model releases', { timeRange: 'week' });

const result = await robot.run();

for (const item of result.searchData['Search Results'].results) {
  console.log(item.title, item.url);
}
```

## Options

| Option | Default | Description |
|---|---|---|
| `mode` | `'discover'` | `'discover'` returns titles, URLs and snippets. `'scrape'` also opens every result and scrapes it |
| `limit` | `10` | Number of results |
| `timeRange` | any time | Only results from the last `'day'`, `'week'`, `'month'` or `'year'` |
| `formats` | `['markdown']` | In scrape mode, what to capture from each result: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` |

## Search modes

### Discover

The default. Fast: returns the search results themselves, without visiting the pages.

```javascript
const robot = await maxun.search('Scraping tools', 'open source web scraping tools', {
  limit: 20,
});

const result = await robot.run();

for (const item of result.searchData['Search Results'].results) {
  console.log(item.position, item.title);
  console.log(item.url);
  console.log(item.description);
}
```

### Scrape

Pass `mode: 'scrape'` to open every result and get its content in the formats you choose.

```javascript
const robot = await maxun.search('Scraping guides', 'how to scrape a website with node.js', {
  mode: 'scrape',
  limit: 5,
  formats: ['markdown', 'links'],
});

const result = await robot.run();

for (const page of result.searchData['Search Results'].results) {
  console.log(page.metadata.url);
  console.log(page.markdown?.slice(0, 300));
}
```

Each scraped result has the formats you asked for plus `metadata` (`url`, `title`, ...) and `searchResult` (the original `position` and title in the search results). A result that could not be opened has an `error` instead.

## Time range

```javascript
const robot = await maxun.search("Today's AI news", 'artificial intelligence', {
  timeRange: 'day',
});
```

Use `'day'`, `'week'`, `'month'` or `'year'`. Leave it out to search any time.

## Examples

### Research a topic

```javascript
const robot = await maxun.search('LLM agents research', 'LLM agent benchmarks', {
  mode: 'scrape',
  limit: 5,
  formats: ['summary'],
});

const result = await robot.run();
for (const page of result.searchData['Search Results'].results) {
  console.log(page.metadata.url, '→', page.summary);
}
```

`summary` uses an LLM. On self-hosted Maxun, also pass `llmProvider`, `llmApiKey` and optionally `llmModel` and `llmBaseUrl`. On Maxun Cloud, leave them out.

### Daily news digest

```javascript
const robot = await maxun.search('Competitor news', 'Acme Corp announcement', {
  timeRange: 'day',
});

await robot.schedule({ runEvery: 1, runEveryUnit: 'DAYS', atTimeStart: '08:00', timezone: 'Asia/Kolkata' });
await robot.addWebhook('https://your-app.com/hooks/news');
```

Every morning Maxun runs the search and sends the results to your webhook.

### Several queries

```javascript
const queries = ['AI automation tools', 'workflow automation software', 'RPA platforms'];

for (const query of queries) {
  const robot = await maxun.search(`Market: ${query}`, query, { timeRange: 'month' });
  const result = await robot.run();
  await saveToDatabase(query, result.searchData['Search Results'].results);
}
```

## Managing search robots

```javascript
const robots = await maxun.search.list();
await robot.setListLimit(25);     // change the number of results
await robot.delete();
```

:::note
Search robots are not deduplicated by name: every `maxun.search(...)` call creates a new robot. Reuse a robot with `await maxun.robots.find(name)` instead of creating it again.
:::

See [Robot Management](./sdk-robot) to run, schedule and manage robots.

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-document" title="Node.js: Extract Data From PDFs & Documents">

# Document

Document robots read files instead of web pages. They can:

- **Extract** specific data from a file, described in plain English.
- **Parse** a file into Markdown, HTML, a list of links or a summary.

Supported files: PDF, DOCX, XLSX, CSV, JPG and PNG.

## Extract

Upload a file and describe the data you want:

```javascript
const robot = await maxun.documents.extract(
  'Invoice reader',
  './invoice.pdf',
  'Invoice number, vendor name, date and total amount',
);

const result = await robot.run();
console.log(result.documentData);
```

```javascript
{
  invoice_number: 'INV-2025-0042',
  vendor_name: 'Acme Corp',
  date: '2025-03-14',
  total_amount: 4250
}
```

The fields you get back follow your prompt.

## Parse

Convert a file into text formats:

```javascript
const robot = await maxun.documents.parse('Annual report', './report.docx', {
  formats: ['markdown', 'summary'],
});

const result = await robot.run();

console.log(result.markdown);
console.log(result.summary);
```

| Format | Read it from |
|---|---|
| `markdown` | `result.markdown` |
| `html` | `result.html` |
| `links` | `result.links` |
| `summary` | `result.summary` |

Leave out `formats` to get all four.

## Passing the file

Pass a path, or a `Buffer`. With a buffer, add `fileName` so Maxun knows the file type:

```javascript

const data = await readFile('invoice.pdf');

const robot = await maxun.documents.extract('Invoice reader', data, 'Invoice number and total', {
  fileName: 'invoice.pdf',
});
```

## Running again

A document robot keeps its file, so you can run it again later. Find it by name:

```javascript
const robot = await maxun.robots.find('Invoice reader');
const result = await robot.run();
```

List all document robots with `await maxun.documents.list()`.

:::note
Document robot names must be unique. Creating one with a name that already exists throws `ConflictError`.
:::

## LLM settings (self-hosted)

`documents.extract` and the `summary` format use an LLM. On Maxun Cloud this is handled for you. On self-hosted Maxun, pass one:

```javascript
const robot = await maxun.documents.extract('Invoice reader', './invoice.pdf', 'Invoice number and total', {
  llmProvider: 'openai',           // 'anthropic', 'openai' or 'ollama'
  llmApiKey: 'your-llm-api-key',
  llmModel: 'gpt-4o-mini',         // optional
});
```

## Complete example

```javascript
import 'dotenv/config';

const maxun = new Maxun();

// Pull the key fields out of an offer letter
const extractor = await maxun.documents.extract(
  'Offer letter fields',
  './offer-letter.pdf',
  'Student name, university, course title and start date',
);
console.log((await extractor.run()).documentData);

// Convert the same letter to Markdown with a summary
const parser = await maxun.documents.parse('Offer letter text', './offer-letter.pdf', {
  formats: ['markdown', 'summary'],
});
const result = await parser.run();
console.log(result.summary);
console.log(result.markdown);
```

See [Robot Management](./sdk-robot) to schedule document robots or add webhooks.

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-monitoring" title="Node.js: Monitor Websites for Changes">

# Monitoring

Monitoring compares every run of a robot with its previous successful run, so you can tell when a page changes: a new price, a new job posting, an updated policy.

It works for **scrape**, **crawl** and **extract** robots (both prompt and selector robots).

## Turn on monitoring

Pass `monitor: true` when you create the robot:

```javascript
// Scrape: watch a page's content
await maxun.scrape('Pricing watch', 'https://example.com/pricing', { formats: ['text'], monitor: true });

// Crawl: watch every page of a site
await maxun.crawl('Docs watch', 'https://docs.example.com', { limit: 50, monitor: true });

// Extract with a prompt: watch specific data
await maxun.extract('Product watch', 'https://shop.example.com/product/42', {
  prompt: 'Product name, price and availability',
  monitor: true,
});

// Extract with selectors
await maxun
  .extract('Listings watch', 'https://shop.example.com/new', { monitor: true })
  .captureList({ selector: '.listing' })
  .build();
```

Or turn it on and off for an existing robot:

```javascript
await robot.setMonitoring(true);
await robot.setMonitoring(false);

console.log(robot.isMonitoring);
```

## Checking for changes

The first run is the baseline. From the second run on, the result says whether anything changed:

```javascript
const result = await robot.run();

if (result.hasChanges) {
  console.log('Changed:', result.changedFormats);
} else {
  console.log('No changes');
}
```

```javascript
console.log(result);
// {
//   runId: '9e1f…',
//   status: 'success',
//   text: '…',
//   hasChanges: true,
//   changedFormats: [ 'text' ]
// }
```

`hasChanges` and `changedFormats` only appear on robots with monitoring turned on.

| Robot | What is compared | `changedFormats` values |
|---|---|---|
| Scrape | The page's `text`, `markdown` and `html` | `text`, `markdown`, `html` |
| Crawl | Every page's `text`, `markdown` and `html`, matched by URL | `text`, `markdown`, `html` |
| Extract | The captured text and lists | `captured-text`, `captured-list` |

## Seeing what changed

`getRunDiff` returns the changes line by line:

```javascript
const diff = await robot.getRunDiff(result.runId);

for (const section of diff.diffs) {
  console.log('Format:', section.format);
  for (const change of section.changes) {
    if (change.added) console.log('+', change.value);
    else if (change.removed) console.log('-', change.value);
  }
}
```

```text
Format: text
- Pro plan: $49 / month
+ Pro plan: $59 / month
```

Limit the diff to one format with the second argument:

```javascript
const diff = await robot.getRunDiff(result.runId, 'markdown');
```

`getRunDiff` works for any run, including scheduled runs:

```javascript
const run = await robot.getLatestRun();
const diff = await robot.getRunDiff(run.runId);
```

### Crawl changes

For crawl robots you also get the pages that were added, removed or changed:

```javascript
const result = await robot.run();

if (result.hasChanges) {
  const pages = result.changedPages;
  console.log('New pages:', pages.added);
  console.log('Removed pages:', pages.removed);
  console.log('Changed pages:', pages.changed);
}
```

The same list is in `diff.pages`.

:::note
Crawl changes are computed by the SDK. `result.hasChanges` is filled in for runs started with `robot.run()`. For scheduled crawl runs, use `getRunDiff`. Webhooks and the Maxun dashboard don't show crawl changes.
:::

## Monitor on a schedule

Monitoring is most useful with a [schedule](./sdk-robot#scheduling) and a [webhook](./sdk-robot#webhooks): Maxun checks the page for you and calls your server after every run.

```javascript
const robot = await maxun.scrape('Pricing watch', 'https://example.com/pricing', {
  formats: ['text'],
  monitor: true,
});

await robot.run();   // first run: the baseline

await robot.schedule({ runEvery: 6, runEveryUnit: 'HOURS' });
await robot.addWebhook('https://your-app.com/hooks/pricing');
```

Later, check the latest scheduled run:

```javascript
const run = await robot.getLatestRun();
const diff = await robot.getRunDiff(run.runId);

if (diff.hasChanges) {
  await notifyTeam(diff);
}
```

## Complete example

```javascript
import 'dotenv/config';

const maxun = new Maxun();

const robot = await maxun.extract('HN top stories watch', 'https://news.ycombinator.com', {
  prompt: 'Title and points of the top 10 stories',
  monitor: true,
});

await robot.run();                     // baseline
const result = await robot.run();      // compared with the baseline

if (!result.hasChanges) {
  console.log('Nothing changed');
} else {
  const diff = await robot.getRunDiff(result.runId);
  for (const section of diff.diffs) {
    for (const change of section.changes) {
      if (change.added || change.removed) {
        console.log(change.added ? '+' : '-', change.value);
      }
    }
  }
}
```

</page>

<page url="https://docs.maxun.dev/sdk/node-sdk/sdk-robot" title="Node.js: Run, Schedule & Manage Robots">

# Robot Management

Every `maxun.scrape(...)`, `maxun.extract(...)`, `maxun.crawl(...)`, `maxun.search(...)` and `maxun.documents...` call returns a **robot**. A robot is saved on your account. You can run it, schedule it, get notified when it finishes and look back at earlier runs.

## Finding robots

```javascript
const all = await maxun.robots.list();              // every robot
const crawlers = await maxun.robots.list('crawl');  // one type
const scrapers = await maxun.scrape.list();         // same thing, per type

const robot = await maxun.robots.get('robot-id');
const pricing = await maxun.robots.find('Pricing page');   // by exact name
```

A robot prints as its id, name and type:

```javascript
console.log(await maxun.robots.list());
// [
//   { id: 'a2e3af0f-fd5b-45cc-b2d0-83e7e5aaf5b3', name: 'Pricing page', type: 'scrape' },
//   { id: '76f3e1c2-9b1a-4c8e-8f0e-2d1c5b7a9e44', name: 'Bookstore', type: 'extract' }
// ]
```

Types are `scrape`, `extract`, `crawl`, `search`, `doc-extract` and `doc-parse`.

Read more about a robot from its properties:

| Property | Description |
|---|---|
| `robot.id` | The robot's id |
| `robot.name` | The robot's name |
| `robot.type` | `scrape`, `extract`, `crawl`, `search`, `doc-extract` or `doc-parse` |
| `robot.url` | The page the robot starts on |
| `robot.formats` | The robot's output formats |
| `robot.isMonitoring` | Whether [monitoring](./sdk-monitoring) is on |
| `robot.getData()` | The full robot record from Maxun |

## Running a robot

```javascript
const result = await robot.run();
```

`run()` waits until the run finishes and returns the result. If the run fails or is aborted, it throws `RunFailedError`.

The result contains the run id, the status and only the outputs the robot produced:

```javascript
console.log(result);
// {
//   runId: 'b05cea93-31d1-4a17-9e6c-e278b4e4abf3',
//   status: 'success',
//   markdown: '# Example Domain\n\n…'
// }
```

Read outputs as properties:

| Property | Filled by |
|---|---|
| `result.runId`, `result.status` | Every run |
| `result.markdown`, `.html`, `.text`, `.links`, `.summary` | Scrape and document parse robots |
| `result.smartQueryResult` | Scrape robots with Smart Queries |
| `result.textData` | `captureText` |
| `result.listData` | `captureList` and prompt extraction |
| `result.crawlData` | Crawl robots, one entry per page |
| `result.searchData` | Search robots |
| `result.documentData` | Document extract robots |
| `result.screenshots` | Screenshot formats and `captureScreenshot` |
| `result.hasChanges`, `.changedFormats` | Robots with [monitoring](./sdk-monitoring) on |

A property for an output the run didn't produce is `undefined` or empty, so it is always safe to read.

### Run options

```javascript
await robot.run({ formats: ['markdown', 'html'] });          // different formats, this run only
await robot.run({ smartQueries: 'Which plan has SSO?' });    // a question, this run only (scrape)
await robot.run({ timeout: 600000 });                        // stop waiting after 10 minutes
```

`timeout` is in milliseconds. The run keeps going on Maxun after `run()` stops waiting. Read it later with `getLatestRun()`.

## Run history

```javascript
const runs = await robot.getRuns();       // newest first
const latest = await robot.getLatestRun();
const run = await robot.getRun('run-id');
```

A run prints as a short summary:

```javascript
console.log(await robot.getRuns());
// [
//   {
//     id: '3faaa1cd-5c3a-4b7e-a0a1-27c8f0b1f9d2',
//     runId: 'bdae3b5a-8f2e-4d71-9c55-0e6f3a2b1c48',
//     robotId: '2c56ce3b-1d9e-4f6a-8b3c-7e5d4a2f1b90',
//     name: 'Pricing page',
//     status: 'success',
//     startedAt: '2026-10-01T00:46:25Z',
//     finishedAt: '2026-10-01T00:47:14Z'
//   }
// ]
```

Times are in UTC (`finishedAt` is `null` while a run is still going). `status` is `queued`, `running`, `success`, `failed`, `aborting` or `aborted`.

Get a run's output with `run.result`. This works for every run, including scheduled ones:

```javascript
const run = await robot.getLatestRun();
if (run?.status === 'success') {
  console.log(run.result.markdown);
}
```

### Aborting a run

```javascript
await robot.abort('run-id');
```

## Scheduling

Run a robot automatically:

```javascript
// Every 6 hours
await robot.schedule({ runEvery: 6, runEveryUnit: 'HOURS' });

// Every day at 9:00 in Kolkata time
await robot.schedule({ runEvery: 1, runEveryUnit: 'DAYS', atTimeStart: '09:00', timezone: 'Asia/Kolkata' });

// Every Monday at 9:00
await robot.schedule({ runEvery: 1, runEveryUnit: 'WEEKS', startFrom: 'MONDAY', atTimeStart: '09:00' });

// On the 1st of every month at 6:30
await robot.schedule({ runEvery: 1, runEveryUnit: 'MONTHS', dayOfMonth: 1, atTimeStart: '06:30' });
```

| Option | Description |
|---|---|
| `runEvery` | How many units between runs |
| `runEveryUnit` | `MINUTES`, `HOURS`, `DAYS`, `WEEKS` or `MONTHS` |
| `timezone` | An IANA time zone, such as `'America/New_York'`. Defaults to `'UTC'` |
| `atTimeStart` | Time of day as `'HH:MM'` for daily, weekly and monthly schedules |
| `atTimeEnd` | Optional end of the time window, as `'HH:MM'` |
| `startFrom` | Day of the week for weekly schedules, such as `'MONDAY'` |
| `dayOfMonth` | Day of the month for monthly schedules |

Check or remove the schedule:

```javascript
const schedule = await robot.getSchedule();
if (schedule) {
  console.log('Next run:', schedule.nextRunAt);
} else {
  console.log('Not scheduled');
}

await robot.unschedule();
```

`getSchedule()` returns `null` when the robot has no schedule.

## Webhooks

Get an HTTP POST to your server every time a run finishes:

```javascript
await robot.addWebhook('https://your-app.com/hooks/maxun');
```

Only failed runs, with more retries:

```javascript
await robot.addWebhook('https://your-app.com/hooks/maxun-alerts', {
  events: ['run_failed'],
  retryAttempts: 5,
});
```

| Option | Default | Description |
|---|---|---|
| `events` | both | `run_completed`, `run_failed` |
| `retryAttempts` | `3` | How many times to retry a failed delivery |
| `retryDelay` | `5` | Seconds before the first retry. The wait grows with each retry |
| `timeout` | `30` | Seconds to wait for your server to respond |

Adding a URL that is already registered updates it instead of adding a duplicate.

```javascript
const hooks = await robot.getWebhooks();                               // [] if none
await robot.removeWebhook('https://your-app.com/hooks/maxun-alerts');  // by URL or id
await robot.removeWebhooks();                                          // remove all
```

### Payload

```json
{
  "event_type": "run_completed",
  "timestamp": "2026-10-01T09:00:42.120Z",
  "webhook_id": "webhook_3f1c…",
  "data": {
    "robot_id": "2c56ce3b-…",
    "run_id": "bdae3b5a-…",
    "robot_name": "Pricing page",
    "status": "success",
    "started_at": "2026-10-01T09:00:03.551Z",
    "finished_at": "2026-10-01T09:00:41.904Z",
    "extracted_data": { "…": "…" }
  }
}
```

## Editing robots

```javascript
await robot.rename('Pricing page (EU)');
await robot.setListLimit(25);           // item limit of a list, crawl or search robot
const copy = await robot.duplicate('https://example.com/eu/pricing');   // same robot, different URL
await robot.refresh();                  // reload the robot from Maxun
```

## Deleting robots

```javascript
await robot.delete();
await maxun.robots.delete('robot-id');
```

## Reusing a robot name

Robot names identify robots in the dashboard. When you create a robot with a name that already exists:

| Robot | Same name |
|---|---|
| Scrape, crawl, prompt extract | Same settings: you get the existing robot. Different settings: `ConflictError` |
| Selector extract | Same URL: you get the existing robot, unchanged, even if your steps differ. The SDK warns you |
| Document | Always `ConflictError` |
| Search | Not checked: a new robot is created every time |

To reuse a robot, find it instead of creating it again:

```javascript

let robot;
try {
  robot = await maxun.scrape('Pricing page', 'https://example.com/pricing');
} catch (error) {
  if (!(error instanceof ConflictError)) throw error;
  robot = await maxun.robots.find('Pricing page');
}
```

## Complete example

```javascript
import 'dotenv/config';

const maxun = new Maxun();

const robot = await maxun
  .extract('Hacker News front page', 'https://news.ycombinator.com')
  .captureList({ selector: 'tr.athing', maxItems: 30 })
  .build();

// Check it works
try {
  const result = await robot.run();
  console.log(`${result.listData.length} stories`);
} catch (error) {
  if (error instanceof RunFailedError) {
    console.error('Run failed:', error.message);
    process.exit(1);
  }
  throw error;
}

// Then run it every morning and get notified
await robot.schedule({ runEvery: 1, runEveryUnit: 'DAYS', atTimeStart: '08:00' });
await robot.addWebhook('https://your-app.com/hooks/maxun');

const schedule = await robot.getSchedule();
console.log('Next run:', schedule?.nextRunAt);
```

## Errors

| Error | When |
|---|---|
| `AuthenticationError` | The API key is missing or invalid |
| `NotFoundError` | The robot or run does not exist |
| `ConflictError` | A robot with that name already exists with different settings |
| `ValidationError` | Maxun rejected the input |
| `RunFailedError` | A run failed or was aborted |

All of them extend `MaxunError`, which has `statusCode` and `details`.

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-overview" title="Python Web Scraping SDK: Install & Setup">

# Maxun Python SDK

The Maxun Python SDK turns websites and documents into structured data from your Python code. You create data scraping robots and run them whenever you need fresh data.

```python
import asyncio
from maxun import Maxun

async def main():
    async with Maxun(api_key="your-api-key") as maxun:
        robot = await maxun.scrape("Maxun", "https://maxun.dev", formats=["markdown"])
        result = await robot.run()
        print(result.markdown)


asyncio.run(main())
```

## Installation

```bash
pip install maxun
```

## Requirements

- Python 3.8+
- A Maxun Cloud account or a self-hosted Maxun instance
- An API key from the [Maxun Dashboard](/api/api)

## Configuration

Pass your API key directly:

```python
from maxun import Maxun

maxun = Maxun(api_key="your-api-key")
```

Or set it in the environment and create `Maxun()` with no arguments:

```bash
MAXUN_API_KEY=your-api-key
MAXUN_TEAM_ID=your-team-uuid                      # optional, Maxun Cloud teams
MAXUN_BASE_URL=http://localhost:8080/api/sdk/     # only for self-hosted Maxun
```

```python
from dotenv import load_dotenv
from maxun import Maxun

load_dotenv()          # only needed if your variables are in a .env file
maxun = Maxun()
```

The SDK connects to Maxun Cloud by default. For a self-hosted instance, set `MAXUN_BASE_URL` or pass `base_url`:

```python
maxun = Maxun(api_key="your-api-key", base_url="http://localhost:8080/api/sdk/")
```

`Maxun` keeps one connection open. Use it with `async with` (as above), or call `await maxun.close()` when you are done.

## Everything starts from `maxun`

Each call takes the **robot name** first, then **what to work on** (a URL, a search query or a file), then any settings as keyword arguments. It returns a [`Robot`](./sdk-robot) saved on your account.

| Call | What the robot does | Read the result from |
|---|---|---|
| [`maxun.scrape(name, url)`](./sdk-scrape) | Turns a page into Markdown, HTML, text, links, a summary or screenshots | `result.markdown`, `result.html`, ... |
| [`maxun.extract(name, url, prompt=...)`](./sdk-extract) | Extracts structured data, described in plain English or with selectors | `result.list_data`, `result.text_data` |
| [`maxun.crawl(name, url)`](./sdk-crawl) | Visits many pages of a website | `result.crawl_data` |
| [`maxun.search(name, query)`](./sdk-search) | Searches the web and optionally scrapes the results | `result.search_data` |
| [`maxun.documents.extract(name, file, prompt)`](./sdk-document) | Extracts data from a PDF, DOCX, XLSX, CSV, JPG or PNG | `result.document_data` |
| [`maxun.documents.parse(name, file)`](./sdk-document) | Converts a document to Markdown, HTML, links or a summary | `result.markdown`, ... |
| [`maxun.robots`](./sdk-robot) | Lists, finds and deletes robots of any type | |

The name is required. It is how the robot appears in the Maxun dashboard.

## Without `async`

Prefer plain function calls? `MaxunSync` has exactly the same methods, without `await`. It also works in Jupyter notebooks.

```python
from maxun import MaxunSync

with MaxunSync(api_key="your-api-key") as maxun:
    robot = maxun.scrape("Example", "https://example.com")
    result = robot.run()
    print(result.markdown)
```

:::note
The examples in these docs use `await`, so they need to run inside an `async` function like the one at the top of this page. With `MaxunSync`, drop the `await`.
:::

## Errors

Every API error is a `MaxunError` with `.status_code` and `.details`. More specific errors:

| Error | When |
|---|---|
| `AuthenticationError` | The API key is missing or invalid |
| `NotFoundError` | The robot or run does not exist |
| `ConflictError` | A robot with that name already exists with different settings |
| `ValidationError` | Maxun rejected the input |
| `RunFailedError` | A run failed or was aborted |

```python
from maxun import ConflictError, MaxunError

try:
    robot = await maxun.scrape("Pricing page", "https://example.com/pricing")
except ConflictError:
    robot = await maxun.robots.find("Pricing page")
except MaxunError as error:
    print(error.status_code, error)
```

## What's next

- [Scrape](./sdk-scrape): turn pages into clean content
- [Extract](./sdk-extract): pull structured data out of pages
- [Crawl](./sdk-crawl): collect content from a whole website
- [Search](./sdk-search): search the web
- [Document](./sdk-document): extract data from and convert documents
- [Robot Management](./sdk-robot): run, schedule and manage robots
- [Monitoring](./sdk-monitoring): get notified when a page changes

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-scrape" title="Python: Scrape Websites to Markdown & HTML">

# Scrape

A scrape robot turns a web page into clean Markdown, HTML, plain text, a list of links, an AI summary or screenshots.

```python
from maxun import Maxun

async with Maxun() as maxun:
    robot = await maxun.scrape("Example page", "https://example.com")
    result = await robot.run()
    print(result.markdown)
```

`maxun.scrape(name, url)` saves the robot on your account. Run it again any time with `await robot.run()`.

## Output formats

Choose what you want back with `formats`. The default is `["markdown"]`.

| Format | Read it from | What you get |
|---|---|---|
| `markdown` | `result.markdown` | The page as clean Markdown, ready for an LLM |
| `html` | `result.html` | The page's HTML |
| `text` | `result.text` | The page's visible text |
| `links` | `result.links` | Every link on the page |
| `summary` | `result.summary` | An AI-written summary of the page |
| `screenshot-visible` | `result.screenshots` | A screenshot of the visible part of the page |
| `screenshot-fullpage` | `result.screenshots` | A screenshot of the whole page |

Ask for as many formats as you need in one robot:

```python
robot = await maxun.scrape(
    "Pricing page",
    "https://example.com/pricing",
    formats=["markdown", "links", "screenshot-fullpage"],
)

result = await robot.run()

print(result.markdown)
print(result.links)
print(result.screenshots)
```

### Summary

`summary` adds a short AI-written summary next to the other formats:

```python
robot = await maxun.scrape(
    "Blog post",
    "https://blog.example.com/post",
    formats=["markdown", "summary"],
)

result = await robot.run()
print(result.summary)
```

## Smart Queries

A Smart Query is a question about the page. Maxun scrapes the page, then an LLM answers your question on every run.

```python
robot = await maxun.scrape(
    "Pricing plans",
    "https://example.com/pricing",
    smart_queries="List every plan with its monthly price.",
)

result = await robot.run()
print(result.smart_query_result)
```

You can also ask a different question for a single run, without changing the robot:

```python
result = await robot.run(smart_queries="Which plan includes SSO?")
print(result.smart_query_result)
```

:::note
`summary` and Smart Queries use an LLM. On Maxun Cloud this is handled for you. On self-hosted Maxun, add the [LLM settings](#llm-settings-self-hosted) below.
:::

## Options

| Option | Default | Description |
|---|---|---|
| `formats` | `["markdown"]` | Output formats (see above) |
| `smart_queries` | none | A question an LLM answers about the page on every run |
| `monitor` | `False` | Compare every run with the previous one. See [Monitoring](./sdk-monitoring) |
| `llm_provider`, `llm_model`, `llm_api_key`, `llm_base_url` | none | Self-hosted Maxun only, see below |

## Running a scrape robot

```python
result = await robot.run()
```

`run()` waits until the page is scraped and returns the result. You can change the formats for a single run:

```python
result = await robot.run(formats=["html"])
print(result.html)
```

If the run fails, `run()` raises `RunFailedError`. See [Robot Management](./sdk-robot) for run history, schedules and webhooks.

## LLM settings (self-hosted)

Self-hosted Maxun has no built-in LLM, so `summary` and Smart Queries need one:

```python
robot = await maxun.scrape(
    "Pricing plans",
    "https://example.com/pricing",
    smart_queries="List every plan with its monthly price.",
    llm_provider="anthropic",        # "anthropic", "openai" or "ollama"
    llm_api_key="your-llm-api-key",  # required for anthropic and openai
    llm_model="claude-sonnet-4-5",   # optional
)
```

:::caution
Do not pass `llm_*` options on Maxun Cloud. Cloud manages the model for you and rejects them.
:::

## Examples

### Feed a RAG pipeline

```python
robot = await maxun.scrape("Docs guide", "https://docs.example.com/guide")
result = await robot.run()

create_embeddings(result.markdown)
```

### Scrape several pages

Give each robot its own name:

```python
pages = {
    "Post: Launch week": "https://blog.example.com/launch-week",
    "Post: Pricing update": "https://blog.example.com/pricing-update",
}

for name, url in pages.items():
    robot = await maxun.scrape(name, url)
    result = await robot.run()
    save_to_database(url, result.markdown)
```

### Use an existing robot

```python
robot = await maxun.robots.find("Pricing plans")   # by name
result = await robot.run()
```

List your scrape robots with `await maxun.scrape.list()`. See [Robot Management](./sdk-robot) for everything else you can do with a robot.

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-extract" title="Python: Extract Structured Data From Websites">

# Extract

An extract robot pulls structured data (product lists, prices, job postings, tables) out of web pages. There are two ways to build one using Maxun SDK:

- **With a prompt**: describe the data in plain English and Maxun builds the robot for you.
- **With selectors**: list the steps and CSS or XPath selectors yourself, for precise and predictable results.

Both return a [`Robot`](./sdk-robot) that you can run as often as you like.

## Extract with a prompt

```python
robot = await maxun.extract(
    "YC companies",
    "https://www.ycombinator.com/companies",
    prompt="Company name, description and batch for the first 15 companies",
)

result = await robot.run()

for company in result.list_data:
    print(company)
```

```python
{'Company name': 'Airbnb', 'Description': 'Book accommodations around the world.', 'Batch': 'W09'}
{'Company name': 'Stripe', 'Description': 'Economic infrastructure for the internet.', 'Batch': 'S09'}
...
```

### Without a URL

On Maxun Cloud, leave the URL out and the agent searches the web for a suitable page first. Self-hosted Maxun requires a URL: [AI Mode](/robot/extract/llm-extraction) reads only the page you give it (see [Agent vs AI Mode](/robot/agent-vs-ai-mode)).

```python
robot = await maxun.extract(
    "Top AI startups",
    prompt="Names and funding of the top 10 AI startups from YCombinator",
)
```

### LLM settings (self-hosted)

On Maxun Cloud, a prompt is all you need. Self-hosted Maxun has no built-in LLM, so pass one:

```python
robot = await maxun.extract(
    "Products",
    "https://shop.example.com",
    prompt="Product names and prices",
    llm_provider="anthropic",        # "anthropic", "openai" or "ollama"
    llm_api_key="your-llm-api-key",  # required for anthropic and openai
    llm_model="claude-sonnet-4-5",   # optional
    llm_base_url=None,               # optional, e.g. your Ollama server
)
```

:::caution
Do not pass `llm_*` options on Maxun Cloud. Cloud manages the model for you and rejects them.
:::

## Extract with selectors

`maxun.extract(name, url)` starts a robot on that page. Chain the steps you want, then finish with `.build()`:

```python
robot = await (
    maxun.extract("Bookstore", "https://books.toscrape.com")
    .capture_text({"Heading": "h1"})
    .capture_list("article.product_pod", max_items=20)
    .build()
)

result = await robot.run()

print(result.text_data)   # {'Heading': 'All products'}
print(result.list_data)   # [{...}, {...}, ...]
```

Selector robots don't use an LLM, so they work the same on Maxun Cloud and self-hosted Maxun.

### Capture text

`capture_text` reads single values. Give each field a name and a selector:

```python
.capture_text({
    "Title": "h1.article-title",
    "Author": ".author-name",
    "Published": "time",
})
```

The values are in `result.text_data`. CSS and XPath selectors both work.

### Capture a list

`capture_list` reads every element that matches a selector. The fields inside each item are detected automatically:

```python
.capture_list("article.product_pod", max_items=50)
```

The items are in `result.list_data`. `max_items` defaults to 100.

### Pagination

Maxun detects pagination automatically. To control it yourself, pass `pagination`:

```python
# Click a "Next" button
.capture_list("article.product_pod", max_items=100,
              pagination={"type": "clickNext", "selector": "li.next a"})

# Click a "Load more" button
.capture_list(".card", pagination={"type": "clickLoadMore", "selector": "button.load-more"})

# Infinite scroll
.capture_list(".feed-item", pagination={"type": "scrollDown"})

# First page only
.capture_list(".result", pagination={"type": "none"})
```

| `type` | Use it for | Needs `selector` |
|---|---|---|
| `clickNext` | A "Next" button or link | Yes |
| `clickLoadMore` | A "Load more" button | Yes |
| `scrollDown` | Infinite scroll | No |
| `scrollUp` | Content that loads when scrolling up | No |
| `none` | Reading only the first page | No |

### Browser actions

Add steps in the order they should happen:

| Step | What it does |
|---|---|
| `.navigate(url)` | Go to another page |
| `.click(selector)` | Click an element |
| `.type(selector, text)` | Type into an input. The text is stored encrypted. |
| `.wait_for(selector, timeout=30000)` | Wait for an element to appear (milliseconds) |
| `.wait(1000)` | Pause (milliseconds) |
| `.scroll(pages=2)` | Scroll down by a number of screen heights |
| `.capture_screenshot("name", full_page=True)` | Take a screenshot, returned in `result.screenshots` |

Give any capture a label with `name="..."`, for example `.capture_list(".product", name="Products")`.

## Examples

### A list across several pages

```python
robot = await (
    maxun.extract("Quotes", "https://quotes.toscrape.com")
    .capture_list(
        "div.quote",
        max_items=50,
        pagination={"type": "clickNext", "selector": "li.next a"},
    )
    .build()
)

result = await robot.run()
print(f"{len(result.list_data)} quotes")
```

### Several pages in one robot

```python
robot = await (
    maxun.extract("Store overview", "https://shop.example.com")
    .capture_text({"Store name": "h1"})
    .navigate("https://shop.example.com/products")
    .capture_list(".product", name="Products")
    .navigate("https://shop.example.com/reviews")
    .capture_list(".review", name="Reviews")
    .build()
)
```

### Log in, then extract

```python
robot = await (
    maxun.extract("Dashboard data", "https://app.example.com/login")
    .type("#email", "you@example.com")
    .type("#password", "your-password")
    .click("button[type=submit]")
    .wait_for(".dashboard")
    .capture_text({"Balance": ".balance", "Plan": ".plan-name"})
    .capture_screenshot("Dashboard")
    .build()
)
```

### Watch for changes

Turn on [monitoring](./sdk-monitoring) to compare every run with the previous one:

```python
robot = await maxun.extract(
    "Product prices",
    "https://shop.example.com/product/42",
    prompt="Product name, price and availability",
    monitor=True,
)
```

For selector robots, pass `monitor=True` before building:

```python
robot = await (
    maxun.extract("Product prices", "https://shop.example.com", monitor=True)
    .capture_list(".product")
    .build()
)
```

## Managing extract robots

```python
robots = await maxun.extract.list()          # all extract robots
robot = await maxun.robots.find("Bookstore") # one robot, by name
await robot.delete()
```

:::note
Building a selector robot with a name and URL that already exist returns the existing robot unchanged, even if your steps are different. The SDK warns you when this happens. Use a new name, or delete the old robot first.
:::

See [Robot Management](./sdk-robot) to run, schedule and manage robots.

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-crawl" title="Python Web Crawler: Crawl Entire Websites">

# Crawl

A crawl robot starts on one page, follows the links it finds and collects the content of every page it visits. Use it to grab a whole documentation site, a blog or a product catalogue.

```python
robot = await maxun.crawl("Example docs", "https://docs.example.com", limit=20)

result = await robot.run()

for page in result.crawl_data:
    print(page["metadata"]["url"])
    print(page.get("markdown", "")[:200])
```

## Options

Every option has a sensible default, so a name and a URL are enough to start.

| Option | Default | Description |
|---|---|---|
| `mode` | `"domain"` | Which links to follow: `"domain"`, `"subdomain"` or `"path"` (see below) |
| `limit` | `50` | Maximum number of pages to crawl |
| `max_depth` | `3` | How many links away from the start page to go |
| `include_paths` | none | Only crawl URLs matching these patterns, e.g. `["/blog/*"]` |
| `exclude_paths` | none | Skip URLs matching these patterns, e.g. `["/admin/*"]` |
| `use_sitemap` | `True` | Also find pages through the site's `sitemap.xml` |
| `follow_links` | `True` | Follow links found on each page |
| `respect_robots` | `True` | Obey the site's `robots.txt` |
| `formats` | `["markdown"]` | What to capture from each page: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` |
| `monitor` | `False` | Compare every run with the previous one. See [Monitoring](./sdk-monitoring) |

```python
robot = await maxun.crawl(
    "Docs guides",
    "https://docs.example.com",
    limit=100,
    max_depth=4,
    include_paths=["/guides/*"],
    exclude_paths=["/guides/archive/*"],
    formats=["markdown", "links"],
)
```

## Crawl modes

`mode` decides how far the crawl may wander from the start URL.

| Mode | Starting at `https://example.com/blog` it crawls |
|---|---|
| `domain` | Any page on `example.com` |
| `subdomain` | Pages on `example.com` and its subdomains, such as `docs.example.com` |
| `path` | Only pages under `example.com/blog` |

```python
robot = await maxun.crawl("Blog", "https://example.com/blog", mode="path")
```

## Reading the results

`result.crawl_data` is a list with one entry per page:

```python
result = await robot.run()

for page in result.crawl_data:
    print(page["metadata"]["url"])
    print(page["metadata"].get("title"))
    print(page.get("markdown"))
```

Each page contains the formats you asked for (`markdown`, `html`, `text`, `links`, ...) plus `metadata` with the page's `url`, `title` and other meta tags. A page that could not be loaded has an `error` instead.

:::tip
Large crawls can take a while. `run()` waits until the crawl finishes. Pass `timeout=600` (seconds) to stop waiting sooner; the crawl keeps going on Maxun and you can read it later with `await robot.get_latest_run()`.
:::

## Examples

### A whole documentation site

```python
robot = await maxun.crawl(
    "Product docs",
    "https://docs.example.com",
    limit=200,
    max_depth=5,
)

result = await robot.run()
docs = {
    page["metadata"]["url"]: page["markdown"]
    for page in result.crawl_data
    if "markdown" in page
}
```

### Only blog posts

```python
robot = await maxun.crawl(
    "Blog posts",
    "https://example.com",
    include_paths=["/blog/*"],
    exclude_paths=["/blog/tag/*", "/blog/page/*"],
    limit=50,
)
```

### Keep a knowledge base up to date

Crawl once a week and see which pages were added, removed or changed:

```python
robot = await maxun.crawl("Help center", "https://help.example.com", limit=100, monitor=True)
await robot.schedule(run_every=1, run_every_unit="WEEKS", start_from="MONDAY", at_time_start="06:00")
```

See [Monitoring](./sdk-monitoring) for how to read the changes.

### Summarize every page

```python
robot = await maxun.crawl("Docs summaries", "https://docs.example.com", formats=["summary"], limit=20)

result = await robot.run()
for page in result.crawl_data:
    print(page["metadata"]["url"], "→", page.get("summary"))
```

`summary` uses an LLM. On self-hosted Maxun, also pass `llm_provider`, `llm_api_key` and optionally `llm_model` and `llm_base_url`. On Maxun Cloud, leave them out.

## Managing crawl robots

```python
robots = await maxun.crawl.list()            # all crawl robots
robot = await maxun.robots.find("Blog posts")
await robot.set_list_limit(500)              # crawl more pages from now on
await robot.delete()
```

See [Robot Management](./sdk-robot) to run, schedule and manage robots.

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-search" title="Python: Web Search & Scrape Results">

# Search

A search robot searches the web (with DuckDuckGo) and returns the results. It can also open each result and scrape its content, so you get search plus page content in one step.

```python
robot = await maxun.search("AI news", "AI model releases", time_range="week")

result = await robot.run()

for item in result.search_data["Search Results"]["results"]:
    print(item["title"], item["url"])
```

## Options

| Option | Default | Description |
|---|---|---|
| `mode` | `"discover"` | `"discover"` returns titles, URLs and snippets. `"scrape"` also opens every result and scrapes it |
| `limit` | `10` | Number of results |
| `time_range` | any time | Only results from the last `"day"`, `"week"`, `"month"` or `"year"` |
| `formats` | `["markdown"]` | In scrape mode, what to capture from each result: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` |

## Search modes

### Discover

The default. Fast: returns the search results themselves, without visiting the pages.

```python
robot = await maxun.search("Scraping tools", "open source web scraping tools", limit=20)

result = await robot.run()

for item in result.search_data["Search Results"]["results"]:
    print(item["position"], item["title"])
    print(item["url"])
    print(item["description"])
```

### Scrape

Pass `mode="scrape"` to open every result and get its content in the formats you choose.

```python
robot = await maxun.search(
    "Scraping guides",
    "how to scrape a website with python",
    mode="scrape",
    limit=5,
    formats=["markdown", "links"],
)

result = await robot.run()

for page in result.search_data["Search Results"]["results"]:
    print(page["metadata"]["url"])
    print(page.get("markdown", "")[:300])
```

Each scraped result has the formats you asked for plus `metadata` (`url`, `title`, ...) and `searchResult` (the original `position` and title in the search results). A result that could not be opened has an `error` instead.

## Time range

```python
robot = await maxun.search("Today's AI news", "artificial intelligence", time_range="day")
```

Use `"day"`, `"week"`, `"month"` or `"year"`. Leave it out to search any time.

## Examples

### Research a topic

```python
robot = await maxun.search(
    "LLM agents research",
    "LLM agent benchmarks",
    mode="scrape",
    limit=5,
    formats=["summary"],
)

result = await robot.run()
for page in result.search_data["Search Results"]["results"]:
    print(page["metadata"]["url"], "→", page.get("summary"))
```

`summary` uses an LLM. On self-hosted Maxun, also pass `llm_provider`, `llm_api_key` and optionally `llm_model` and `llm_base_url`. On Maxun Cloud, leave them out.

### Daily news digest

```python
robot = await maxun.search("Competitor news", "Acme Corp announcement", time_range="day")
await robot.schedule(run_every=1, run_every_unit="DAYS", at_time_start="08:00", timezone="Asia/Kolkata")
await robot.add_webhook("https://your-app.com/hooks/news")
```

Every morning Maxun runs the search and sends the results to your webhook.

### Several queries

```python
queries = ["AI automation tools", "workflow automation software", "RPA platforms"]

for query in queries:
    robot = await maxun.search(f"Market: {query}", query, time_range="month")
    result = await robot.run()
    save_to_database(query, result.search_data["Search Results"]["results"])
```

## Managing search robots

```python
robots = await maxun.search.list()
await robot.set_list_limit(25)     # change the number of results
await robot.delete()
```

:::note
Search robots are not deduplicated by name: every `maxun.search(...)` call creates a new robot. Reuse a robot with `await maxun.robots.find(name)` instead of creating it again.
:::

See [Robot Management](./sdk-robot) to run, schedule and manage robots.

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-document" title="Python: Extract Data From PDFs & Documents">

# Document

Document robots read files instead of web pages. They can:

- **Extract** specific data from a file, described in plain English.
- **Parse** a file into Markdown, HTML, a list of links or a summary.

Supported files: PDF, DOCX, XLSX, CSV, JPG and PNG.

## Extract

Upload a file and describe the data you want:

```python
robot = await maxun.documents.extract(
    "Invoice reader",
    "./invoice.pdf",
    "Invoice number, vendor name, date and total amount",
)

result = await robot.run()
print(result.document_data)
```

```python
{
    "invoice_number": "INV-2025-0042",
    "vendor_name": "Acme Corp",
    "date": "2025-03-14",
    "total_amount": 4250
}
```

The fields you get back follow your prompt.

## Parse

Convert a file into text formats:

```python
robot = await maxun.documents.parse(
    "Annual report",
    "./report.docx",
    formats=["markdown", "summary"],
)

result = await robot.run()

print(result.markdown)
print(result.summary)
```

| Format | Read it from |
|---|---|
| `markdown` | `result.markdown` |
| `html` | `result.html` |
| `links` | `result.links` |
| `summary` | `result.summary` |

Leave out `formats` to get all four.

## Passing the file

Pass a path, or the file's bytes. With bytes, add `file_name` so Maxun knows the file type:

```python
with open("invoice.pdf", "rb") as f:
    data = f.read()

robot = await maxun.documents.extract(
    "Invoice reader",
    data,
    "Invoice number and total",
    file_name="invoice.pdf",
)
```

## Running again

A document robot keeps its file, so you can run it again later. Find it by name:

```python
robot = await maxun.robots.find("Invoice reader")
result = await robot.run()
```

List all document robots with `await maxun.documents.list()`.

:::note
Document robot names must be unique. Creating one with a name that already exists raises `ConflictError`.
:::

## LLM settings (self-hosted)

`documents.extract` and the `summary` format use an LLM. On Maxun Cloud this is handled for you. On self-hosted Maxun, pass one:

```python
robot = await maxun.documents.extract(
    "Invoice reader",
    "./invoice.pdf",
    "Invoice number and total",
    llm_provider="openai",           # "anthropic", "openai" or "ollama"
    llm_api_key="your-llm-api-key",
    llm_model="gpt-4o-mini",         # optional
)
```

## Complete example

```python
import asyncio

from dotenv import load_dotenv
from maxun import Maxun

load_dotenv()


async def main():
    async with Maxun() as maxun:
        # Pull the key fields out of an offer letter
        extractor = await maxun.documents.extract(
            "Offer letter fields",
            "./offer-letter.pdf",
            "Student name, university, course title and start date",
        )
        print((await extractor.run()).document_data)

        # Convert the same letter to Markdown with a summary
        parser = await maxun.documents.parse(
            "Offer letter text",
            "./offer-letter.pdf",
            formats=["markdown", "summary"],
        )
        result = await parser.run()
        print(result.summary)
        print(result.markdown)


asyncio.run(main())
```

See [Robot Management](./sdk-robot) to schedule document robots or add webhooks.

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-monitoring" title="Python: Monitor Websites for Changes">

# Monitoring

Monitoring compares every run of a robot with its previous successful run, so you can tell when a page changes: a new price, a new job posting, an updated policy.

It works for **scrape**, **crawl** and **extract** robots.

## Turn on monitoring

Pass `monitor=True` when you create the robot:

```python
# Scrape: watch a page's content
robot = await maxun.scrape("Pricing watch", "https://example.com/pricing", formats=["text"], monitor=True)

# Crawl: watch every page of a site
robot = await maxun.crawl("Docs watch", "https://docs.example.com", limit=50, monitor=True)

# Extract with a prompt: watch specific data
robot = await maxun.extract(
    "Product watch",
    "https://shop.example.com/product/42",
    prompt="Product name, price and availability",
    monitor=True,
)

# Extract with selectors
robot = await (
    maxun.extract("Listings watch", "https://shop.example.com/new", monitor=True)
    .capture_list(".listing")
    .build()
)
```

Or turn it on and off for an existing robot:

```python
await robot.set_monitoring(True)
await robot.set_monitoring(False)

print(robot.is_monitoring)
```

## Checking for changes

The first run is the baseline. From the second run on, the result says whether anything changed:

```python
result = await robot.run()

if result.has_changes:
    print("Changed:", result.changed_formats)
else:
    print("No changes")
```

```python
>>> result
{'runId': '9e1f…', 'status': 'success', 'text': '…', 'hasChanges': True, 'changedFormats': ['text']}
```

`hasChanges` and `changedFormats` only appear on robots with monitoring turned on.

| Robot | What is compared | `changed_formats` values |
|---|---|---|
| Scrape | The page's `text`, `markdown` and `html` | `text`, `markdown`, `html` |
| Crawl | Every page's `text`, `markdown` and `html`, matched by URL | `text`, `markdown`, `html` |
| Extract | The captured text and lists | `captured-text`, `captured-list` |

## Seeing what changed

`get_run_diff` returns the changes line by line:

```python
diff = await robot.get_run_diff(result.run_id)

for section in diff["diffs"]:
    print("Format:", section["format"])
    for change in section["changes"]:
        if change["added"]:
            print("+", change["value"])
        elif change["removed"]:
            print("-", change["value"])
```

```text
Format: text
- Pro plan: $49 / month
+ Pro plan: $59 / month
```

Limit the diff to one format with `format`:

```python
diff = await robot.get_run_diff(result.run_id, format="markdown")
```

`get_run_diff` works for any run, including scheduled runs:

```python
run = await robot.get_latest_run()
diff = await robot.get_run_diff(run.run_id)
```

### Crawl changes

For crawl robots you also get the pages that were added, removed or changed:

```python
result = await robot.run()

if result.has_changes:
    pages = result.changed_pages
    print("New pages:", pages["added"])
    print("Removed pages:", pages["removed"])
    print("Changed pages:", pages["changed"])
```

The same list is in `diff["pages"]`.

:::note
Crawl changes are computed by the SDK. `result.has_changes` is filled in for runs started with `robot.run()`. For scheduled crawl runs, use `get_run_diff`. Webhooks and the Maxun dashboard don't show crawl changes.
:::

## Monitor on a schedule

Monitoring is most useful with a [schedule](./sdk-robot#scheduling) and a [webhook](./sdk-robot#webhooks): Maxun checks the page for you and calls your server after every run.

```python
robot = await maxun.scrape("Pricing watch", "https://example.com/pricing", formats=["text"], monitor=True)

await robot.run()   # first run: the baseline

await robot.schedule(run_every=6, run_every_unit="HOURS")
await robot.add_webhook("https://your-app.com/hooks/pricing")
```

Later, check the latest scheduled run:

```python
run = await robot.get_latest_run()
diff = await robot.get_run_diff(run.run_id)

if diff["hasChanges"]:
    notify_team(diff)
```

## Complete example

```python
import asyncio

from dotenv import load_dotenv
from maxun import Maxun

load_dotenv()


async def main():
    async with Maxun() as maxun:
        robot = await maxun.extract(
            "HN top stories watch",
            "https://news.ycombinator.com",
            prompt="Title and points of the top 10 stories",
            monitor=True,
        )

        await robot.run()                 # baseline
        result = await robot.run()        # compared with the baseline

        if not result.has_changes:
            print("Nothing changed")
            return

        diff = await robot.get_run_diff(result.run_id)
        for section in diff["diffs"]:
            for change in section["changes"]:
                if change["added"] or change["removed"]:
                    print("+" if change["added"] else "-", change["value"])


asyncio.run(main())
```

</page>

<page url="https://docs.maxun.dev/sdk/python-sdk/sdk-robot" title="Python: Run, Schedule & Manage Robots">

# Robot Management

Every `maxun.scrape(...)`, `maxun.extract(...)`, `maxun.crawl(...)`, `maxun.search(...)` and `maxun.documents...` call returns a **robot**. A robot is saved on your account. You can run it, schedule it, get notified when it finishes and look back at earlier runs.

## Finding robots

```python
robots = await maxun.robots.list()                # every robot
robots = await maxun.robots.list(type="crawl")    # one type
robots = await maxun.scrape.list()                # same thing, per type

robot = await maxun.robots.get("robot-id")
robot = await maxun.robots.find("Pricing page")   # by exact name
```

A robot prints as its id, name and type:

```python
>>> await maxun.robots.list()
[
    {"id": "a2e3af0f-fd5b-45cc-b2d0-83e7e5aaf5b3", "name": "Pricing page", "type": "scrape"},
    {"id": "76f3e1c2-9b1a-4c8e-8f0e-2d1c5b7a9e44", "name": "Bookstore", "type": "extract"}
]
```

Types are `scrape`, `extract`, `crawl`, `search`, `doc-extract` and `doc-parse`.

Read more about a robot from its attributes:

| Attribute | Description |
|---|---|
| `robot.id` | The robot's id |
| `robot.name` | The robot's name |
| `robot.type` | `scrape`, `extract`, `crawl`, `search`, `doc-extract` or `doc-parse` |
| `robot.url` | The page the robot starts on |
| `robot.formats` | The robot's output formats |
| `robot.is_monitoring` | Whether [monitoring](./sdk-monitoring) is on |
| `robot.get_data()` | The full robot record from Maxun |

## Running a robot

```python
result = await robot.run()
```

`run()` waits until the run finishes and returns the result. If the run fails or is aborted, it raises `RunFailedError`.

The result contains the run id, the status and only the outputs the robot produced:

```python
>>> result
{'runId': 'b05cea93-31d1-4a17-9e6c-e278b4e4abf3', 'status': 'success', 'markdown': '# Example Domain\n\n…'}
```

Read outputs as attributes:

| Attribute | Filled by |
|---|---|
| `result.run_id`, `result.status` | Every run |
| `result.markdown`, `.html`, `.text`, `.links`, `.summary` | Scrape and document parse robots |
| `result.smart_query_result` | Scrape robots with Smart Queries |
| `result.text_data` | `capture_text` |
| `result.list_data` | `capture_list` and prompt extraction |
| `result.crawl_data` | Crawl robots, one entry per page |
| `result.search_data` | Search robots |
| `result.document_data` | Document extract robots |
| `result.screenshots` | Screenshot formats and `capture_screenshot` |
| `result.has_changes`, `.changed_formats` | Robots with [monitoring](./sdk-monitoring) on |

An attribute for an output the run didn't produce is `None` or empty, so it is always safe to read.

### Run options

```python
result = await robot.run(formats=["markdown", "html"])          # different formats, this run only
result = await robot.run(smart_queries="Which plan has SSO?")   # a question, this run only (scrape)
result = await robot.run(timeout=600)                            # stop waiting after 600 seconds
```

With `timeout`, the run keeps going on Maxun after `run()` stops waiting. Read it later with `get_latest_run()`.

## Run history

```python
runs = await robot.get_runs()           # newest first
run = await robot.get_latest_run()
run = await robot.get_run("run-id")
```

A run prints as a short summary:

```python
>>> await robot.get_runs()
[
    {
        "id": "3faaa1cd-5c3a-4b7e-a0a1-27c8f0b1f9d2",
        "runId": "bdae3b5a-8f2e-4d71-9c55-0e6f3a2b1c48",
        "robotId": "2c56ce3b-1d9e-4f6a-8b3c-7e5d4a2f1b90",
        "name": "Pricing page",
        "status": "success",
        "startedAt": "2026-10-01T00:46:25Z",
        "finishedAt": "2026-10-01T00:47:14Z"
    }
]
```

Times are in UTC. `status` is `queued`, `running`, `success`, `failed`, `aborting` or `aborted`.

Get a run's output with `run.result`. This works for every run, including scheduled ones:

```python
run = await robot.get_latest_run()
if run.status == "success":
    print(run.result.markdown)
```

### Aborting a run

```python
await robot.abort("run-id")
```

## Scheduling

Run a robot automatically:

```python
# Every 6 hours
await robot.schedule(run_every=6, run_every_unit="HOURS")

# Every day at 9:00 in Kolkata time
await robot.schedule(run_every=1, run_every_unit="DAYS", at_time_start="09:00", timezone="Asia/Kolkata")

# Every Monday at 9:00
await robot.schedule(run_every=1, run_every_unit="WEEKS", start_from="MONDAY", at_time_start="09:00")

# On the 1st of every month at 6:30
await robot.schedule(run_every=1, run_every_unit="MONTHS", day_of_month=1, at_time_start="06:30")
```

| Option | Description |
|---|---|
| `run_every` | How many units between runs |
| `run_every_unit` | `MINUTES`, `HOURS`, `DAYS`, `WEEKS` or `MONTHS` |
| `timezone` | An IANA time zone, such as `"America/New_York"`. Defaults to `"UTC"` |
| `at_time_start` | Time of day as `"HH:MM"` for daily, weekly and monthly schedules |
| `at_time_end` | Optional end of the time window, as `"HH:MM"` |
| `start_from` | Day of the week for weekly schedules, such as `"MONDAY"` |
| `day_of_month` | Day of the month for monthly schedules |

Check or remove the schedule:

```python
schedule = await robot.get_schedule()
if schedule:
    print("Next run:", schedule["nextRunAt"])
else:
    print("Not scheduled")

await robot.unschedule()
```

`get_schedule()` returns `None` when the robot has no schedule.

## Webhooks

Get an HTTP POST to your server every time a run finishes:

```python
await robot.add_webhook("https://your-app.com/hooks/maxun")
```

Only failed runs, with more retries:

```python
await robot.add_webhook(
    "https://your-app.com/hooks/maxun-alerts",
    events=["run_failed"],
    retry_attempts=5,
)
```

| Option | Default | Description |
|---|---|---|
| `events` | both | `run_completed`, `run_failed` |
| `retry_attempts` | `3` | How many times to retry a failed delivery |
| `retry_delay` | `5` | Seconds before the first retry. The wait grows with each retry |
| `timeout` | `30` | Seconds to wait for your server to respond |

Adding a URL that is already registered updates it instead of adding a duplicate.

```python
hooks = await robot.get_webhooks()                                  # [] if none
await robot.remove_webhook("https://your-app.com/hooks/maxun-alerts")   # by URL or id
await robot.remove_webhooks()                                       # remove all
```

### Payload

```json
{
  "event_type": "run_completed",
  "timestamp": "2026-10-01T09:00:42.120Z",
  "webhook_id": "webhook_3f1c…",
  "data": {
    "robot_id": "2c56ce3b-…",
    "run_id": "bdae3b5a-…",
    "robot_name": "Pricing page",
    "status": "success",
    "started_at": "2026-10-01T09:00:03.551Z",
    "finished_at": "2026-10-01T09:00:41.904Z",
    "extracted_data": { "…": "…" }
  }
}
```

## Editing robots

```python
await robot.rename("Pricing page (EU)")
await robot.set_list_limit(25)          # item limit of a list, crawl or search robot
copy = await robot.duplicate("https://example.com/eu/pricing")   # same robot, different URL
await robot.refresh()                   # reload the robot from Maxun
```

## Deleting robots

```python
await robot.delete()
await maxun.robots.delete("robot-id")
```

## Reusing a robot name

Robot names identify robots in the dashboard. When you create a robot with a name that already exists:

| Robot | Same name |
|---|---|
| Scrape, crawl, prompt extract | Same settings: you get the existing robot. Different settings: `ConflictError` |
| Selector extract | Same URL: you get the existing robot, unchanged, even if your steps differ. The SDK warns you |
| Document | Always `ConflictError` |
| Search | Not checked: a new robot is created every time |

To reuse a robot, find it instead of creating it again:

```python
from maxun import ConflictError

try:
    robot = await maxun.scrape("Pricing page", "https://example.com/pricing")
except ConflictError:
    robot = await maxun.robots.find("Pricing page")
```

## Complete example

```python
import asyncio

from dotenv import load_dotenv
from maxun import Maxun, RunFailedError

load_dotenv()


async def main():
    async with Maxun() as maxun:
        robot = await (
            maxun.extract("Hacker News front page", "https://news.ycombinator.com")
            .capture_list("tr.athing", max_items=30)
            .build()
        )

        # Check it works
        try:
            result = await robot.run()
            print(f"{len(result.list_data)} stories")
        except RunFailedError as error:
            print("Run failed:", error)
            return

        # Then run it every morning and get notified
        await robot.schedule(run_every=1, run_every_unit="DAYS", at_time_start="08:00")
        await robot.add_webhook("https://your-app.com/hooks/maxun")

        schedule = await robot.get_schedule()
        print("Next run:", schedule["nextRunAt"])


asyncio.run(main())
```

## Errors

| Error | When |
|---|---|
| `AuthenticationError` | The API key is missing or invalid |
| `NotFoundError` | The robot or run does not exist |
| `ConflictError` | A robot with that name already exists with different settings |
| `ValidationError` | Maxun rejected the input |
| `RunFailedError` | A run failed or was aborted |

All of them are subclasses of `MaxunError`, which has `.status_code` and `.details`.

</page>

<page url="https://docs.maxun.dev/cli/cli-overview" title="Maxun CLI: Install & Quick Start">

# Maxun CLI

The Maxun CLI lets you create robots, trigger runs, and fetch extracted data directly from your terminal — without opening the dashboard. It's fully compatible with both the **Cloud version** and **Self-hosted / OSS version** of Maxun.

## Requirements

- Node.js 18.0.0 or higher
- A Maxun account and API key from [app.maxun.dev](https://app.maxun.dev) (for Cloud) or your local instance (for Self-hosted).


## Installation

```bash
npm install -g maxun-cli
```

## Authentication

### Cloud Version

```bash
maxun login --api-key <your-api-key>
```

### Self-hosted / OSS Version

```bash
maxun login --api-url http://localhost:8080 --api-key <your-api-key>
```

Your config is stored at `~/.maxun/config.json` and used automatically for all subsequent commands.

You can also set these via environment variables:

```bash
export MAXUN_API_KEY=your-api-key
export MAXUN_API_URL=http://localhost:8080
```

## Quick Start

```bash
# Authenticate
maxun login --api-key <your-api-key>

# Create an AI robot from a prompt
maxun robots extract -p "Extract trending repositories from Github" -n "Github Trends"

# Run the robot
maxun run <robot-id>

# Check status and credits
maxun status
```

</page>

<page url="https://docs.maxun.dev/cli/cli-robots" title="CLI: Create & Manage Robots">

# Robots

The `maxun robots` command group lets you create, inspect, and manage robots from the terminal.

## On this page

* [Manage Robots](#manage-robots)
  * [List Robots](#list-robots)
  * [Get, Duplicate, and Delete Robots](#get-duplicate-and-delete-robots)
* [Create Robots](#create-robots)
  * [AI extraction](#ai-extraction)
  * [Scrape](#scrape)
  * [Crawl](#crawl)
  * [Search](#search)
  * [Document extract](#document-extract-doc-extract)
  * [Document parse](#document-parse-doc-parse)

## Manage Robots

### List Robots

`maxun robots list` returns all robots as JSON by default. Use `--table` for a formatted view:

```bash
maxun robots list
maxun robots list --table
```

### Get, Duplicate, and Delete Robots

| Command                                       | Description                                   |
| --------------------------------------------- | --------------------------------------------- |
| `maxun robots get <id>`                       | Get details for a specific robot              |
| `maxun robots duplicate <id> --url <new-url>` | Duplicate a robot with a different target URL |
| `maxun robots delete <id>`                    | Delete a robot                                |

## Create Robots

Choose a command based on what you want to do:

| Command       | Use case                                                        |
| ------------- | --------------------------------------------------------------- |
| `extract`     | Extract structured data from a webpage using AI                 |
| `scrape`      | Scrape a single webpage                                         |
| `crawl`       | Crawl multiple pages on a website                               |
| `search`      | Find webpages using a search query                              |
| `doc-extract` | Extract structured fields from a document using AI              |
| `doc-parse`   | Convert a document into Markdown, HTML, or links without an LLM |


### AI Extraction

Create an AI-powered robot from a natural language prompt. Maxun uses LLMs to infer the extraction schema from the page.

```bash
maxun robots extract -p <prompt> [options]
```

| Option | Description |
|--------|-------------|
| `-p, --prompt <text>` | Natural language description of what to extract (required) |
| `-u, --url <url>` | Target URL for the robot. Optional on Maxun Cloud (the agent finds the site); required for self-hosted Maxun |
| `-n, --name <name>` | Robot name |
| `--provider <provider>` | LLM provider: `huggingface`, `openrouter` (default: `huggingface`) |
| `--model <model>` | LLM model name |
| `--api-key <key>` | LLM API key |

**Example:**
```bash
maxun robots extract \
  -p "Extract all product names and prices" \
  -u "https://example.com/shop" \
  -n "Shop Extractor"
```

### Scrape

Create a single-page scraping robot.

```bash
maxun robots scrape <url> [options]
```

| Option | Description |
|--------|-------------|
| `-n, --name <name>` | Robot name |
| `-f, --format <fmt>` | Output formats: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` (comma-separated, default: `markdown`) |
| `-p, --prompt <text>` | Smart Queries: LLM prompt to analyze the page and perform actions post scraping |

**Example:**
```bash
maxun robots scrape https://example.com -f markdown,text -n "Example Scraper"
```

### Crawl

Create a multi-page crawler robot.

```bash
maxun robots crawl <url> [options]
```

| Option | Description |
|--------|-------------|
| `-n, --name <name>` | Robot name |
| `-f, --format <fmt>` | Output formats (comma-separated, default: `markdown`) |
| `--limit <n>` | Max pages to crawl (default: 10) |
| `--include <paths>` | Include path patterns (comma-separated) |
| `--exclude <paths>` | Exclude path patterns (comma-separated) |

**Example:**
```bash
maxun robots crawl https://docs.example.com --limit 20 --include "/docs/*" -n "Docs Crawler"
```

### Search

Create a search-based robot.

```bash
maxun robots search <query> [options]
```

| Option | Description |
|--------|-------------|
| `-n, --name <name>` | Robot name |
| `-f, --format <fmt>` | Output formats (comma-separated) |
| `--limit <n>` | Max search results (default: 10) |
| `--mode <mode>` | `discover` (URLs + metadata) or `scrape` (full content, default: `discover`) |

**Example:**
```bash
maxun robots search "Latest AI news" --mode discover --limit 10 -n "AI News"
```

### Document Extract (doc-extract)

Create a robot that extracts structured fields from a document using AI.

```bash
maxun robots doc-extract <document> [options]
```

| Option | Description |
|--------|-------------|
| `-p, --prompt <prompt>` | What to extract in natural language (required) |
| `-n, --name <name>` | Robot name |
| `--model <model>` | Ollama Cloud model override |

**Example:**
```bash
maxun robots doc-extract invoice.pdf \
  --prompt "Extract invoice number, vendor name, issue date, and total amount" \
  --name "Invoice Extractor"
```

### Document Parse (doc-parse)

Create a robot that converts a document into Markdown, HTML, and/or extracted links — no LLM, free.

```bash
maxun robots doc-parse <document> [options]
```

| Option | Description |
|--------|-------------|
| `-f, --formats <formats>` | Output formats, comma-separated: `markdown`, `html`, `links` (required) |
| `-n, --name <name>` | Robot name |

**Example:**
```bash
maxun robots doc-parse banner.png --formats "markdown" --name "PNG Banner Parser"
```

</page>

<page url="https://docs.maxun.dev/cli/cli-run" title="CLI: Run Robots & Get Results">

# Running Robots

Use `maxun run` to trigger a robot execution and receive results directly in your terminal.

## Basic Usage

```bash
maxun run <robot-id>
```

Triggers the robot and streams the extracted data to stdout as JSON once the run completes.

## Options

| Option | Description |
|--------|-------------|
| `-f, --format <fmt>` | Override output formats for this run: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` (comma-separated) |
| `-t, --table` | Display results in a table (best suited for Discovery Search results) |

## Examples

### Standard run

```bash
maxun run <robot-id>
```

### Override output format

```bash
maxun run <robot-id> -f html,markdown
```

### Get an AI-generated summary

```bash
maxun run <robot-id> -f summary
```

Combine with other formats:

```bash
maxun run <robot-id> -f summary,markdown
```

### View search results as a table

```bash
maxun run <robot-id> --table
```

## Output

Results are printed to stdout as JSON by default — pipe-friendly for scripting:

```bash
maxun run <robot-id> | jq '.[].price'
```

After a run completes, the data is also stored and retrievable via:

```bash
maxun runs get <robot-id> <run-id>
```

</page>

<page url="https://docs.maxun.dev/cli/cli-runs" title="CLI: List Runs & Export Data to CSV/JSON">

# Runs & Data

The `maxun runs` command group lets you list past runs for a robot and retrieve extracted data in multiple formats.

## List Runs

```bash
maxun runs list <robot-id>
```

Returns recent runs as JSON by default.

| Option | Description |
|--------|-------------|
| `--limit <n>` | Max results to return (default: 10) |
| `-t, --table` | Display in table format |

**Example:**
```bash
maxun runs list <robot-id> --table
```

## Get Run Data

```bash
maxun runs get <robot-id> <run-id>
```

Fetches the extracted output for a specific run.

| Option | Description |
|--------|-------------|
| `-f, --format <fmt>` | Output format: `json` (default), `csv`, `table` |
| `-o, --output <file>` | Save output to a file |
| `--pretty` | Pretty print JSON output |

**Examples:**

```bash
# Default JSON output
maxun runs get <robot-id> <run-id>

# Pretty JSON
maxun runs get <robot-id> <run-id> --pretty

# Table view (for search discovery results)
maxun runs get <robot-id> <run-id> -f table

# Export to CSV
maxun runs get <robot-id> <run-id> -f csv -o results.csv

# Save JSON to file
maxun runs get <robot-id> <run-id> -o data.json
```

## Abort a Run

```bash
maxun runs abort <robot-id> <run-id>
```

Aborts a run that is currently in progress.

</page>

<page url="https://docs.maxun.dev/cli/cli-reference" title="Maxun CLI Command Reference">

# Command Reference

Full reference for all `maxun` CLI commands.

## Auth

| Command | Description |
|---------|-------------|
| `maxun login --api-url <url> --api-key <key>` | Authenticate with your Maxun instance and API key |
| `maxun logout` | Clear stored credentials |
| `maxun status` | Show authentication status, plan, and credits |
| `maxun credits` | Show remaining credit balance |

## Robots

| Command | Description |
|---------|-------------|
| `maxun robots list` | List all robots (JSON by default, `--table` for formatted view) |
| `maxun robots extract -p <prompt> [-u <url>] [--provider <p>] [--model <m>]` | Create an AI robot from a natural language prompt |
| `maxun robots scrape <url> [-f <formats>]` | Create a single-page scraping robot; `-f` sets output formats: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` |
| `maxun robots crawl <url> [-f <formats>]` | Create a multi-page crawling robot; `-f` sets per-page output formats |
| `maxun robots search <query> [-f <formats>]` | Create a search-based robot; `-f` sets per-result output formats (scrape mode only) |
| `maxun robots get <id>` | Get details for a specific robot |
| `maxun robots delete <id>` | Delete a robot |
| `maxun robots duplicate <id> --url <url>` | Duplicate a robot with a new target URL |

## Execution

| Command | Description |
|---------|-------------|
| `maxun run <id>` | Run a robot and return results (JSON by default) |
| `maxun run <id> --table` | Run and display results in table format |
| `maxun run <id> -f <formats>` | Override output formats for this run: `markdown`, `html`, `text`, `links`, `summary`, `screenshot-visible`, `screenshot-fullpage` (comma-separated) |

## Runs & Data

| Command | Description |
|---------|-------------|
| `maxun runs list <robot-id>` | List recent runs for a robot |
| `maxun runs get <robot-id> <run-id>` | Get output of a specific run |
| `maxun runs get ... -f csv -o file.csv` | Export run output as CSV |
| `maxun runs abort <robot-id> <run-id>` | Abort an in-progress run |

## Global Flags

| Flag | Description |
|------|-------------|
| `-v, --version` | Print CLI version |
| `-h, --help` | Show help for any command |

## Environment Variables

| Variable | Description |
|----------|-------------|
| `MAXUN_API_KEY` | API key — overrides stored config |
| `MAXUN_API_URL` | API base URL — defaults to https://app.maxun.dev |

## Configuration File

Credentials are stored at `~/.maxun/config.json`:

```json
{
  "apiKey": "your-api-key",
  "apiUrl": "http://localhost:8080"
}
```

</page>

<page url="https://docs.maxun.dev/usecases/content_aggregation" title="Automate Content Aggregation From Any Website">

# Content Aggregation

Maxun automates the collection of news articles, blog posts, research papers and product specifications from many websites. Keep your content platforms updated or build comprehensive knowledge bases without manual copy-pasting.

## What You Can Collect

- Article titles, authors, dates and summaries
- Full article content as clean Markdown
- Research papers and their metadata
- Product specifications from manufacturer sites

## How to Set It Up

1. **Collect lists of content.** Use an [Extract robot](/robot/extract/robot-actions) to capture headlines and links from a news or blog index.
2. **Get the full content.** [Scrape](/robot/scrape) individual pages, or [Crawl](/robot/crawl) a whole blog or documentation site into Markdown, ready for AI tools.
3. **Discover new sources.** Use [Search](/robot/search/search-introduction) to find recent content on a topic, filtered by day, week or month.
4. **Keep the feed fresh.** [Schedule](/robot-schedule) runs and deliver results with [webhooks](/api/webhooks), or build a RAG knowledge base with [LlamaIndex](/integrations/llamaindex).

## In Action: Curate Top Stories from Medium Automatically

<video controls width="100%">
  <source src="https://www.maxun.dev/Medium_2x.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

</page>

<page url="https://docs.maxun.dev/usecases/market_research" title="Automate Market Research With Web Data">

# Market Research

Good market research needs fresh data from many sources. Maxun collects product specifications, customer reviews, pricing and competitor activity from e-commerce and review sites, and keeps it up to date automatically.

## What You Can Collect

- Product specifications and feature lists
- Customer reviews and ratings
- Pricing across retailers
- Competitor product launches and catalog changes

## How to Set It Up

1. **Pick the right robot.** Use [Extract](/robot/extract/robot-actions) for structured lists such as products or reviews, [Crawl](/robot/crawl) to collect a whole site or section, and [Search](/robot/search/search-introduction) to discover relevant pages for a topic.
2. **Describe the data in plain English.** [AI Mode](/robot/extract/llm-extraction) builds the robot from a prompt like "Extract product names, prices and ratings".
3. **Keep it fresh.** [Schedule](/robot-schedule) runs and turn on [Monitoring](/monitoring) to track changes over time.
4. **Analyze it.** Export to CSV or JSON, sync to [Google Sheets](/integrations/gsheet), or feed it into an AI pipeline with [LangChain](/integrations/langchain) or [LlamaIndex](/integrations/llamaindex).

## In Action: Track Competitor Prices on Amazon Automatically

<video controls width="100%">
  <source src="https://www.maxun.dev/Amazon_2x.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

</page>

<page url="https://docs.maxun.dev/usecases/lead_generation" title="Lead Generation Scraping: Build Targeted Lists">

# Lead Generation

Maxun extracts company and contact information from online directories, social media and business listings to build targeted lead lists for sales and marketing campaigns. Maxun handles the data extraction, so you can focus on outreach.

## What You Can Collect

- Company names, websites and descriptions
- Locations and categories
- Public contact details listed on directory pages
- Profile information from business listings

## How to Set It Up

1. **Extract the list.** Create an [Extract robot](/robot/extract/robot-actions) on a directory or listing page and capture every entry with pagination.
2. **Get the details.** Use [Deep Extraction](/deep-extraction) on Maxun Cloud to open each listing and capture the details from its own page.
3. **Scale it.** [Duplicate the robot](/robot/extract/robot-duplicate) for other categories or locations with the same page structure.
4. **Send leads to your tools.** Sync to [Google Sheets](/integrations/gsheet) or [Airtable](/integrations/airtable), or trigger an [n8n](/integrations/n8n) workflow for enrichment and outreach.

> Only collect publicly available data, follow each website's terms of service and comply with data protection laws such as GDPR.

## In Action: Build a Targeted Investor List from OpenVC in Minutes

<video controls width="100%">
  <source src="https://www.maxun.dev/OpenVC_2x.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

</page>

<page url="https://docs.maxun.dev/usecases/ecommerce_automation" title="E-Commerce Price & Stock Monitoring">

# E-Commerce Price & Stock Monitoring

Prices and stock levels change constantly across online stores. Maxun monitors product pages for you and turns them into structured data you can act on: competitor prices, discounts, availability and new products.

## What You Can Collect

- Product names, prices, discounts and currencies
- Stock and availability status
- Ratings and review counts
- New products added to a category

## How to Set It Up

1. **Extract the product data.** Create an [Extract robot](/robot/extract/robot-actions) on a category or search results page and capture the product list with pagination. Or describe the fields in plain English with [AI Mode](/robot/extract/llm-extraction).
2. **Reuse it across stores and categories.** [Duplicate the robot](/robot/extract/robot-duplicate) for other pages with the same structure.
3. **Schedule it.** Run the robot [daily or hourly](/robot-schedule).
4. **Get alerts on changes.** Turn on [Monitoring](/monitoring) to see exactly what changed since the last run.
5. **Send the data where you need it.** Push results to [Google Sheets](/integrations/gsheet), [Airtable](/integrations/airtable), [n8n](/integrations/n8n) or your own app via [webhooks](/api/webhooks).

## In Action: Monitor Pricing & Stock on Reliance Digital 24/7

<video controls width="100%">
  <source src="https://www.maxun.dev/Reliance_Digital_2x.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

</page>

<page url="https://docs.maxun.dev/usecases/real_estate" title="Scrape Real Estate Listings & Rental Prices">

# Real Estate Data Extraction

Maxun extracts property listings, agent contact details, rental prices and market trends from real estate portals. Power your real estate analytics, investment strategies or lead generation with fresh data.

## What You Can Collect

- Listing titles, prices and price per square foot
- Location, size, bedrooms and amenities
- Rental prices and availability
- Agent and broker details

## How to Set It Up

1. **Extract the listings.** Create an [Extract robot](/robot/extract/robot-actions) on a search results page and capture every listing with pagination.
2. **Get full property details.** Use [Deep Extraction](/deep-extraction) on Maxun Cloud to visit each listing page and capture its details.
3. **Cover more locations.** [Duplicate the robot](/robot/extract/robot-duplicate) for other cities or neighborhoods.
4. **Track the market.** [Schedule](/robot-schedule) daily runs and turn on [Monitoring](/monitoring) to spot new listings and price drops.

## In Action: Find Your Next Investment Property on Housing.com

<video controls width="100%">
  <source src="https://www.maxun.dev/Housing_2x.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

</page>

<page url="https://docs.maxun.dev/usecases/academic_research" title="Web Scraping for Academic Research">

# Academic Research

Maxun collects data from scientific journals, online databases and public archives for your research projects. Automate data gathering for literature reviews, statistical analysis and trend identification, without writing code.

## What You Can Collect

- Paper titles, authors, abstracts and publication dates
- Citation and journal information
- Datasets and tables published on public websites
- Full text of open-access articles as Markdown

## How to Set It Up

1. **Extract search results.** Create an [Extract robot](/robot/extract/robot-actions) on a journal or database search page and capture every result with pagination.
2. **Parse documents.** Use [Document robots](/robot/document) to extract fields from PDFs or convert them to Markdown.
3. **Build datasets.** Export results as CSV or JSON, or sync them to [Google Sheets](/integrations/gsheet) for analysis.
4. **Stay up to date.** [Schedule](/robot-schedule) runs to collect new publications automatically.

## In Action: Automate Your Literature Review with Springer Papers

<video controls width="100%">
  <source src="https://www.maxun.dev/Springer_2x.mp4" type="video/mp4" />
  Your browser does not support the video tag.
</video>

</page>

<page url="https://docs.maxun.dev/installation/docker" title="Install Maxun With Docker Compose">

# Docker Compose

This guide will help you install and run Maxun with Docker Compose. Docker Compose is a tool that helps you define and share multi-container applications. You don't need to install each dependency used by Maxun separately as Docker Compose will automatically do it for you.

## Prerequisites

| Software    | Download Link                                                                 |
|-------------|-------------------------------------------------------------------------------|
| Docker Desktop | [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)                           |



## Installation Steps
1. Create a root folder for your project (e.g. 'maxun')

2. Generate secure secrets by running these commands and saving the output:

```bash
openssl rand -base64 48  # For JWT_SECRET
openssl rand -base64 48  # For SESSION_SECRET
openssl rand -hex 32     # For ENCRYPTION_KEY
```

3. Create a file named `.env` in the root folder of the project

4. Copy all content of <a href="https://github.com/getmaxun/maxun/blob/master/ENVEXAMPLE">example env file</a> to your `.env` file.

5. Replace the placeholder values for `JWT_SECRET`, `SESSION_SECRET`, and `ENCRYPTION_KEY` with the secrets you generated in step 2.

6. Create a file named `docker-compose.yml` in the root folder of the project

7. Copy all content from <a href="https://github.com/getmaxun/maxun/blob/develop/docker-compose.yml">project's docker-compose.yml</a> to your `docker-compose.yml` file.

8. Run

```
docker-compose up -d
```

![Docker Compose successful start](docker.png)

You can access the frontend at `PUBLIC_URL` (eg: http://localhost:5173/) and backend at `BACKEND_URL` (eg: http://localhost:8080/)

To set the `PUBLIC_URL` & `BACKEND_URL` environment variables, refer <a href="/installation/environment_variables">Environment Variables</a> section.

</page>

<page url="https://docs.maxun.dev/installation/local" title="Run Maxun Locally (Node.js Development Setup)">

# Local Setup

This guide will help you install and run Maxun on your local system.

## Prerequisites

| Software    | Download Link                                                                 |
|-------------|-------------------------------------------------------------------------------|
| Node.js (18 or above)     | [Download Node.js](https://nodejs.org/en/download/)                           |
| PostgreSQL  | [Download PostgreSQL](https://www.postgresql.org/download/)                   |
| MinIO       | [Download MinIO](https://min.io/download)                                     |

## Installation Steps

1. Clone the repository
```
git clone https://github.com/getmaxun/maxun
```
2. Create a `.env` file in the project root folder (maxun in this case)
3. Copy the content from the <a href="https://github.com/getmaxun/maxun/blob/master/ENVEXAMPLE">example env file</a> to your `.env`
4. Ensure you have the prerequisites installed on your system.
5. Install dependencies
```
# change directory to the project root
cd maxun

# install dependencies
npm install

# change directory to maxun-core to install dependencies
cd maxun-core 
npm install

# get back to the root directory
cd ..
```
6. Browser Setup - Choose one of the following options:

   **Option A: Use the Browser Docker Service** (Recommended)
   
   If you want to use the browser Docker service, configure these environment variables in your `.env` file:
   - `BROWSER_WS_PORT`: WebSocket port for browser connections (default: 3001)
   - `BROWSER_HEALTH_PORT`: Health check port for the browser service (default: 3002)
   - `BROWSER_WS_HOST`: Browser service host (set to `localhost` for local setup)
   
   Then run the browser service using Docker:
   ```bash
   docker-compose up -d browser
   ```

   **Option B: Install Chromium Locally**
   
   If you prefer not to use the browser Docker service, install Chromium:
   ```bash
   npx playwright install --with-deps chromium
   ```

6. Run the command below in the root directory of the project

```
# start frontend and backend together
npm run start
```

You can access the frontend at `PUBLIC_URL` (eg: http://localhost:5173/) and backend at `BACKEND_URL` (eg: http://localhost:8080/)

To set the `PUBLIC_URL` & `BACKEND_URL` environment variables, refer <a href="/installation/environment_variables">Environment Variables</a> section.

</page>

<page url="https://docs.maxun.dev/installation/environment_variables" title="Maxun Environment Variables Reference (.env)">

# Environment Variables
It is important to configure all environment variables labeled as "Mandatory" to ensure Maxun operates smoothly.

## Generating Secure Secrets

The following variables must be set to unique, randomly generated values for each deployment:

```bash
openssl rand -base64 48  # For JWT_SECRET
openssl rand -base64 48  # For SESSION_SECRET
openssl rand -hex 32     # For ENCRYPTION_KEY
```

## Variable Reference

| Variable              | Mandatory | Description                                                                                  | If Not Set                                                   |
|-----------------------|-----------|----------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| `BACKEND_PORT`            | Yes       | Port to run backend on. Needed for Docker setup                                          | Default value: 8080 |
| `FRONTEND_PORT`            | Yes       | Port to run frontend on. Needed for Docker setup                                        | Default value: 5173 |
| `BACKEND_URL`            | Yes       | URL to run backend on.                                                                    | Default value: http://localhost:8080 |
| `VITE_BACKEND_URL`            | Yes       | URL used by frontend to connect to backend                                           | Default value: http://localhost:8080 |
| `PUBLIC_URL`            | Yes       | URL to run frontend on.                                                                    | Default value: http://localhost:5173 |
| `VITE_PUBLIC_URL`            | Yes       | URL used by backend to connect to frontend                                           | Default value: http://localhost:5173 |
| `JWT_SECRET`          | Yes       | Secret key for signing JWTs. Generate with `openssl rand -base64 48`               | Server will refuse to start.                            |
| `DB_NAME`             | Yes       | Name of the Postgres database to connect to.                                                 | Database connection will fail.                               |
| `DB_USER`             | Yes       | Username for Postgres database authentication.                                               | Database connection will fail.                               |
| `DB_PASSWORD`         | Yes       | Password for Postgres database authentication.                                               | Database connection will fail.                               |
| `DB_HOST`             | Yes       | Host address where the Postgres database server is running.                                  | Database connection will fail.                               |
| `DB_PORT`             | Yes       | Port number used to connect to the Postgres database server.                                 | Database connection will fail.                               |
| `DB_SSL`              | No        | Set to `true` to enable SSL for the database connection. Required when using Neon or any SSL-enforced PostgreSQL host. | Defaults to `false`. SSL will not be used. |
| `ENCRYPTION_KEY`      | Yes       | Key for encrypting sensitive data. Generate with `openssl rand -hex 32`                                | Encryption functionality will not work.                      |
| `SESSION_SECRET`      | Yes       | Secret for signing session cookies. Generate with `openssl rand -base64 48`                                         | Sessions won't persist across restarts.  |
| `MINIO_ENDPOINT`      | Yes       | Endpoint URL for MinIO, to store Robot Run Screenshots.                                      | Connection to MinIO storage will fail.                       |
| `MINIO_PORT`          | Yes       | Port number for MinIO service.                                                               | Connection to MinIO storage will fail.                       |
| `MINIO_CONSOLE_PORT`          | No       | Port number for MinIO WebUI service. Needed for Docker setup.                         | Cannot access MinIO Web UI. |
| `MINIO_ACCESS_KEY`    | Yes       | Access key for authenticating with MinIO.                                                    | MinIO authentication will fail.                              |
| `MINIO_SECRET_KEY`    | Yes       | Secret key for authenticating with MinIO. Generate with `openssl rand -base64 24`           | MinIO authentication will fail.                              |
| `MINIO_PUBLIC_URL`    | No        | Public base URL browsers use to reach stored screenshots and files, e.g. `https://storage.example.com`. Set it when MinIO is behind a reverse proxy. | Files are served from the MinIO endpoint and port. |
| `REDIS_HOST`          | Yes       | Host of the Redis server. Use `redis` (or your Redis container name) in Docker.               | Redis connection will fail.                                  |
| `REDIS_PORT`          | Yes       | Port of the Redis server.                                                                    | Default value: 6379 |
| `REDIS_PASSWORD`      | No        | Password for a password-protected Redis instance.                                            | Redis connects without authentication. |
| `GOOGLE_CLIENT_ID`    | No       | Client ID for Google OAuth, used for Google Sheet integration authentication.                 | Google login will not work.                                  |
| `GOOGLE_CLIENT_SECRET`| No       | Client Secret for Google OAuth.                                                              | Google login will not work.                                  |
| `GOOGLE_REDIRECT_URI` | No       | Redirect URI for handling Google OAuth responses.                                            | Google login will not work.                                  |
| `AIRTABLE_CLIENT_ID` | No       | Client ID for Airtable, used for Airtable integration authentication.                         | Airtable login will not work.  |
| `AIRTABLE_REDIRECT_URI` | No    | Redirect URI for handling Airtable OAuth responses.                                           | Airtable login will not work.  |
| `BROWSER_WS_PORT`     | Yes       | Port for WebSocket connections to the browser service (used for CDP connections).            | Default value: 3001 |
| `BROWSER_HEALTH_PORT` | Yes       | Port for browser service health checks.                                                      | Default value: 3002 |
| `BROWSER_WS_HOST`     | Yes       | Host address for the browser service. Set to `browser` for Docker, `localhost` for local.   | Browser service connections will fail. |
| `MAXUN_TELEMETRY`     | No        | Enables anonymous usage telemetry. Keeping it enabled helps us understand how the product is used and assess the impact of any new changes. Please keep it enabled. | Telemetry data will not be collected. |
| `ANTHROPIC_API_KEY`   | No        | API key for Anthropic Claude. Used by AI Mode when Anthropic is the LLM provider and no key is entered in Model settings. | AI Mode with Anthropic Claude will not work without a key in Model settings. |
| `OPENAI_API_KEY`      | No        | API key for OpenAI GPT. Used by AI Mode when OpenAI is the LLM provider and no key is entered in Model settings. | AI Mode with OpenAI GPT will not work without a key in Model settings. |
| `OLLAMA_BASE_URL`     | No        | Ollama server for AI Mode. Use `http://host.docker.internal:11434` when Maxun runs in Docker and Ollama on the host, or `https://ollama.com` for Ollama Cloud. | Default value: `http://localhost:11434` |
| `OLLAMA_API_KEY`      | No        | API key sent to the Ollama server. Required for Ollama Cloud; a local Ollama ignores it. | Ollama Cloud requests are rejected. |
| `OLLAMA_MODEL`        | No        | Ollama model used by AI Mode when none is chosen in Model settings, e.g. `gpt-oss:20b` on Ollama Cloud. | Default value: `llama3.2` |

</page>

<page url="https://docs.maxun.dev/self-host" title="Self-Host Maxun in Production (Docker, NGINX, HTTPS)">

# Self-Host Maxun in Production

This guide deploys Maxun on your own server with **Docker Compose**, behind an **NGINX reverse proxy** with **HTTPS**. Maxun is only exposed on `localhost`, and NGINX serves it securely on your own domain (for example `maxun.my.domain`).

For a quick local install to try Maxun, see [Docker Compose](/installation/docker) instead. For a full list of settings, see [Environment Variables](/installation/environment_variables).

## Prerequisites

- A server with **Docker** and **Docker Compose**
- A web server for the reverse proxy (this guide uses **NGINX**)
- **SSL certificates** (for example from Let's Encrypt or ZeroSSL)
- A domain or subdomain for Maxun, such as `maxun.my.domain`

## 1. Create the Folders

Create a folder for Maxun and the sub-folders used to persist data between container restarts and updates:

```bash
cd /home/$USER/Docker/
mkdir -p maxun/{db,minio,redis}
cd maxun
```

## 2. Create the `.env` File

**Option A: generate it automatically (recommended).** From the root of a cloned [Maxun repository](https://github.com/getmaxun/maxun), run the generator. It writes `.env` into the current directory, so run it from the folder your `docker-compose.yml` will live in. It needs `openssl` and bash (on Windows, use WSL or Git Bash).

```bash
bash docs/generate-env.sh
```

**Option B: write it yourself.** Generate the secrets:

```bash
openssl rand -base64 48   # JWT_SECRET
openssl rand -base64 24   # DB_PASSWORD
openssl rand -hex 32      # ENCRYPTION_KEY (must be 64 hex characters)
openssl rand -base64 48   # SESSION_SECRET
openssl rand -base64 24   # MINIO_SECRET_KEY
```

Then create `.env` and paste in the values:

```dotenv
NODE_ENV=production
JWT_SECRET=<output of first command>
DB_NAME=maxun
DB_USER=postgres
DB_PASSWORD=<output of second command>
DB_HOST=postgres
DB_PORT=5432
ENCRYPTION_KEY=<output of third command>
SESSION_SECRET=<output of fourth command>
MINIO_ENDPOINT=minio
MINIO_PORT=9000
MINIO_CONSOLE_PORT=9001
MINIO_ACCESS_KEY=minio
MINIO_SECRET_KEY=<output of fifth command>
REDIS_HOST=maxun-redis
REDIS_PORT=6379
REDIS_PASSWORD=
BACKEND_PORT=8080
FRONTEND_PORT=5173
BACKEND_URL=https://maxun.my.domain
PUBLIC_URL=https://maxun.my.domain
VITE_BACKEND_URL=https://maxun.my.domain
VITE_PUBLIC_URL=https://maxun.my.domain
MAXUN_TELEMETRY=true
```

> **Note:** Don't put a literal `$` in any value. Docker Compose reads it as a variable and silently drops it, leaving you with a shorter secret. Double it (`$$`) to escape it.

Whichever option you use, check the file and set `BACKEND_URL`, `PUBLIC_URL`, `VITE_BACKEND_URL` and `VITE_PUBLIC_URL` to your domain, and change any ports you need.

## 3. Create `docker-compose.yml`

Use the production compose file from the [full self-hosting guide](https://github.com/getmaxun/maxun/blob/develop/docs/self-hosting-docker.md). It runs five services:

| Service | Image | Purpose |
|---|---|---|
| `postgres` | `postgres:17` | Database for robots, runs and users |
| `redis` | `redis:7` | In-memory data store used by the backend |
| `minio` | `minio/minio` | Storage for screenshots and files |
| `backend` | `getmaxun/maxun-backend:latest` | API, robots and browser automation, bound to `127.0.0.1:8080` |
| `frontend` | `getmaxun/maxun-frontend:latest` | Web app, bound to `127.0.0.1:5173` |

The backend and frontend only listen on `127.0.0.1`, so Maxun is not reachable from the internet until you configure the reverse proxy.

## 4. Start Maxun

```bash
sudo docker compose up -d
```

Wait about 30 seconds for all services to start. Maxun is now available locally at `http://localhost:5173`.

## 5. Configure NGINX and HTTPS

Point your domain at the server and configure NGINX to:

- Redirect HTTP to HTTPS and serve your SSL certificate
- Proxy backend paths (`/auth`, `/storage`, `/record`, `/workflow`, `/robot`, `/proxy`, `/api-docs`, `/api`, `/webhook`, `/socket.io`) to the backend port (`8080`), with WebSocket upgrade headers
- Proxy everything else (`/`) to the frontend port (`5173`)
- Add security headers and gzip compression

A complete, ready-to-edit configuration is available in the Maxun repository: [nginx.conf](https://github.com/getmaxun/maxun/blob/develop/docs/nginx.conf). Replace `maxun.my.domain` and the certificate paths with your own, then reload NGINX.

## Next Steps

- [Environment Variables](/installation/environment_variables): all available settings
- [Upgrading](/installation/upgrade): update to the latest version
- [BYOP](/byop): add your own proxies to reduce blocking
- [Cloud vs Open Source](/cloud-vs-oss): what Maxun Cloud adds on top of the self-hosted edition

The full guide is available on GitHub: [Self-Hosting Maxun with Docker](https://github.com/getmaxun/maxun/blob/develop/docs/self-hosting-docker.md).

</page>

<page url="https://docs.maxun.dev/installation/upgrade" title="Upgrade Self-Hosted Maxun (Docker & Local)">

# Upgrading

Upgrading Maxun is straightforward, whether you're running it locally or using Docker Compose. Follow the instructions below based on your setup.


## Upgrading with Docker Compose

1.  **Navigate to your Project Directory**

    Open your terminal or command prompt and change your current directory to your Maxun project folder (e.g., `maxun`).

    ```bash
    cd maxun
    ```

2.  **Stop Running Containers**

    First, stop any currently running Maxun containers.

    ```bash
    docker-compose down
    ```

3.  **Remove Old Docker Images (Optional but Recommended)**

    To free up disk space and ensure you're using the freshest images without any caching issues, remove your old backend and frontend images.

    > **Note:** If you're on **Windows**, run these commands in **PowerShell**, **Git Bash**, or **WSL** (Windows Subsystem for Linux), not Command Prompt (CMD).

    ```bash
    docker rmi getmaxun/maxun-frontend:latest getmaxun/maxun-backend:latest
    ```

4.  **Pull Latest Docker Images**

    Pull the newest Docker images for your backend and frontend services.

    ```bash
    docker-compose pull backend frontend
    ```

5.  **Start Maxun**

    Launch Maxun using the updated Docker images.

    ```bash
    docker-compose up -d
    ```
---

## Upgrading with Local Setup

If you've set up Maxun directly on your machine, follow these steps to upgrade to the latest version.

1.  **Navigate to your Project Directory**

    Open your terminal or command prompt and change your current directory to your Maxun project folder (e.g., `maxun`).

    ```bash
    cd maxun
    ```

2.  **Pull Latest Changes**

    Fetch the most recent updates from the `master` branch of the repository.

    ```bash
    git pull origin master
    ```

3.  **Install Dependencies**

    Update your project's dependencies to ensure everything is compatible with the new code.

    ```bash
    npm install
    ```

4.  **Start Maxun**

    Launch Maxun.

    ```bash
    npm run start
    ```

</page>

<page url="https://docs.maxun.dev/cloud-vs-oss" title="Maxun Cloud vs Self-Hosted Open Source">

Maxun is available in two ways: **Maxun Cloud** and **Self-Hosted Open Source**.

Both provide the core Maxun platform for turning websites into structured data  -  including extraction, crawling, scraping, search, monitoring, APIs, SDKs, CLI, and MCP.

The difference is what happens when you need to run those workflows **reliably at scale**.

**Maxun Cloud adds managed browser infrastructure, anti-bot capabilities, advanced extraction, pre-built robots, collaboration, and operational features  -  so you can focus on what data you want rather than running the infrastructure required to collect it.**

Self-hosting gives you complete control over the infrastructure and is ideal when you want to run Maxun yourself, customize the deployment, or bring your own proxies and infrastructure.

## At a Glance

| Capability                                | Maxun Cloud |     Self-Hosted Open Source    |
| :---------------------------------------- | :---------: | :----------------------------: |
| Core extraction, scraping & crawling      |      ✅      |                ✅               |
| API, SDK & CLI                            |      ✅      |                ✅               |
| MCP                                       |      ✅      |                ✅               |
| Search                                    |      ✅      |                ✅               |
| Monitoring                                |      ✅      |                ✅               |
| Document extraction                       |      ✅      |                ✅               |
| Deep Extraction                           |      ✅      |                ❌               |
| Robots from a prompt                      | ✅ [Agent](/robot/agent) (URL optional, finds the page, applies filters & sorting) | ✅ [AI Mode](/robot/extract/llm-extraction) (reads the page you enter) |
| Extract behind login                      |      ✅ Managed Chrome Extension | ✅ Manual login via Recorder Mode |
| Automatic adaptation to website changes   |      ✅      |                ❌               |
| Managed stealth & anti-bot infrastructure |      ✅      |                ❌ Bring Your Own Proxy |
| Automatic proxy rotation                  |      ✅      |                ❌               |
| Captcha bypass                            |      ✅      |                ❌               |
| Long-running extraction infrastructure    |      ✅      | Depends on your infrastructure |
| Teams & collaboration                     |      ✅      |                ❌               |
| Managed notifications                     |      ✅      |                ❌               |
| Priority support                          |      ✅      |                ❌               |

## What You Get With Maxun Cloud

### Managed infrastructure for difficult websites

Cloud is designed for extraction workloads where simply running a browser is not enough.

Maxun Cloud manages the infrastructure needed to run browser-based extraction reliably, including:

* **Stealth enabled by default** to reduce blocking on websites with bot detection
* **Automatic proxy rotation**
* **Managed anti-bot infrastructure**
* **Captcha bypass**
* Long-running extraction jobs
* Infrastructure that scales with your workloads

You don't need to source proxies, configure browser infrastructure, or build your own system for handling websites that actively try to block automation.

With self-hosting, you control this infrastructure yourself. You can bring your own proxy and configure your own stealth and deployment setup.

> **In short:** Cloud handles the infrastructure required to make extraction work reliably. Self-hosting gives you the tools to build and operate that infrastructure yourself.

[Learn more about Stealth →](/stealth)
[Learn more about BYOP →](/byop)

## Advanced Extraction

Cloud also unlocks extraction capabilities that are not available in the open-source edition.

### Deep Extraction

**Deep Extraction** lets a robot automatically follow items from a list or search page into their individual detail pages and extract data from both.

For example:

```text
Product listing
    ↓
Product 1 → details
Product 2 → details
Product 3 → details
...
    ↓
Complete structured dataset
```

This is useful for product catalogs, business directories, job boards, marketplaces, and other websites where the data you need is spread across listing and detail pages.

Deep Extraction is available exclusively in Maxun Cloud.

[Learn more about Deep Extraction →](/deep-extraction)

### Automatic adaptation

Cloud can automatically adapt extraction when a website's layout or structure changes.

Self-hosted Maxun also supports retraining extract robots, allowing you to update a robot when the website changes.

The difference is that Cloud provides the managed adaptive extraction experience, while self-hosted deployments give you the tools to retrain and maintain your robots yourself.

### Extract behind login

Maxun Cloud provides a managed Chrome Extension for securely sharing an authenticated browser session with Maxun.

This lets you authenticate to a website in your own browser and share the session with Maxun without manually entering or exposing your credentials to the platform. This is particularly useful for extracting data from authenticated websites while keeping the login flow simple.

With Self-Hosted Maxun, authenticated extraction uses the standard Recorder Mode flow. You manually log in through the recorder and the authenticated session is used for the robot.

Cloud is therefore better suited to recurring or production authenticated extraction where you want a managed login and session-sharing experience.


## AI-Powered Extraction

Both editions build Extract robots from a plain-English prompt, in different ways:

* **[Agents](/robot/agent)** on Cloud take a prompt and, optionally, a URL. An agent can find the right website itself, move to the page that holds the data, apply the site's own search, filters and sorting, answer a question about the data on every run, and check its result before it is saved. The model is managed for you.
* **[AI Mode](/robot/extract/llm-extraction)** in the open-source edition takes a URL and a prompt and extracts from that page only. It never opens other pages, except the page's own *Next* control when you ask for more items than the first page shows. You bring your own LLM: Anthropic, OpenAI or an OpenAI-compatible server, or Ollama.

[Agent vs AI Mode →](/robot/agent-vs-ai-mode)

## Monitoring

Both Cloud and Self-Hosted Maxun support monitoring for supported extraction workflows.

Maxun Cloud additionally supports Crawl Monitoring, allowing you to monitor entire websites and automatically detect changes across crawled pages.

Cloud also provides email alerts when monitored data changes.

Self-hosted Maxun currently does not include a built-in alerting system, so you can run monitoring workflows but must handle notifications yourself.

| Monitoring capability                      | Maxun Cloud | Self-Hosted Open Source |
| :----------------------------------------- | :---------: | :---------------------: |
| Full page monitoring (Scrape)              |      ✅      |            ✅            |
| Specific data monitoring (Extract)         |      ✅      |            ✅            |
| Prompt monitoring (AI Extract)             |      ✅      |            ✅            |
| Website / multiple-page monitoring (Crawl) |      ✅      |            ❌            |
| Alerts                                     |      ✅      |            ❌            |


## Collaboration & Teams

Maxun Cloud is built for teams as well as individual users.

Cloud provides:

* Multiple workspaces
* Shared robots and runs
* Team members and roles
* Admin, Member, and Viewer permissions
* Shared API configuration
* Centralized usage across a team

Self-hosted Maxun is primarily designed around your own deployment and infrastructure. You are responsible for implementing additional organization-level infrastructure and access controls around your deployment.

[Learn more about Teams →](/teams-management)

## Integrations


Maxun integrates with both data tools and AI development frameworks.

Available integrations include:

| Integration       | What it enables                                                    |
| :---------------- | :----------------------------------------------------------------- |
| **Google Sheets** | Sync extracted data directly into Google Sheets                    |
| **n8n**           | Send extracted data directly into n8n workflows                    |
| **Airtable**      | Sync extracted data directly into an Airtable Base                 |
| **LangChain**     | Build AI-powered web scraping chains and agents with the Maxun SDK |
| **LangGraph**     | Build stateful, multi-step AI workflows with Maxun                 |
| **Vercel AI SDK** | Integrate Maxun into React and Next.js AI applications             |
| **OpenAI SDK**    | Use Maxun for web data retrieval through OpenAI function calling   |
| **Mastra**        | Build AI agent workflows with Maxun                                |
| **LlamaIndex**    | Build RAG applications using Maxun                                 |
| **OpenClaw**      | List and run Maxun robots from messaging apps                      |
| **Claude Code**   | List and run Maxun robots directly from your terminal              |

Cloud additionally provides managed integrations and infrastructure around these workflows, while self-hosting lets you control how everything is deployed and connected.

## What Self-Hosting Gives You

Self-hosting is not a limited version of Maxun. The open-source edition contains the core Maxun platform and gives you complete control over how it runs.

You can:

* Run Maxun on your own infrastructure
* Use the dashboard, API, SDK, CLI, and MCP
* Scrape, crawl, extract, search, and monitor websites
* Configure your own LLM provider
* Bring your own proxy infrastructure
* Customize the deployment
* Keep your data and infrastructure under your control

The trade-off is that **you operate the infrastructure yourself**.

That means you are responsible for things such as proxy configuration, scaling, browser infrastructure, anti-bot handling, deployment, monitoring, and operational reliability.

[Learn more about Self-Hosting →](/self-host)

## Which Should You Use?

### Choose Maxun Cloud if you want to:

* Start extracting data without managing infrastructure
* Extract from websites with bot detection
* Run extraction at scale
* Use automatic proxy rotation and managed anti-bot infrastructure
* Use Stealth without configuring it yourself
* Use Deep Extraction
* Use pre-trained Auto Robots
* Build more advanced AI extraction workflows
* Run long-running extraction jobs
* Monitor websites and receive managed notifications
* Collaborate with a team
* Centralized workspace management
* Get priority support

### Choose Self-Hosted Open Source if you want to:

* Run Maxun on your own infrastructure
* Have complete control over deployment and data
* Customize Maxun for your environment
* Bring your own proxy infrastructure
* Configure your own LLM providers
* Operate Maxun entirely within your own infrastructure
* Contribute to and extend the open-source project

## The Simple Difference

**Maxun Cloud is Maxun with the infrastructure and advanced capabilities managed for you.**

**Self-Hosted Open Source gives you the core Maxun platform and complete control over how you run it.**

Both let you turn websites into structured data. Cloud is designed for users who want to **focus on the data and workflows**, while self-hosting is designed for users who want to **own and operate the infrastructure themselves**.

</page>

<page url="https://docs.maxun.dev/faq-robot" title="Maxun FAQ: Pricing, Robots, API, Logins & More">

<head>
  <script type="application/ld+json">
    {`{"@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{"@type": "Question", "name": "What is Maxun?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun is an open-source web data platform that turns any website into clean, structured data for your apps, workflows and AI agents. You can scrape, crawl, extract, search and monitor the web through the REST API, Node.js and Python SDKs, CLI and MCP server, or through a no-code dashboard where anyone can build robots by clicking on a page or describing what they need in plain English."}}, {"@type": "Question", "name": "Who is Maxun for?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun is built for both developers and non-technical teams. Developers and AI engineers use the API, SDKs, CLI and MCP server to power products and agents. Growth, operations and research teams use the no-code dashboard to build and schedule robots without writing code. Both work on the same platform, so teams can hand work back and forth."}}, {"@type": "Question", "name": "What can I build with Maxun?", "acceptedAnswer": {"@type": "Answer", "text": "Teams use Maxun for price and competitor monitoring, lead generation, market research, RAG pipelines, AI agents that need live web data, real estate and job aggregators, review tracking and content aggregation. If the data is on a website, you can turn it into a dataset, a scheduled feed or an API."}}, {"@type": "Question", "name": "How does Maxun work?", "acceptedAnswer": {"@type": "Answer", "text": "You tell Maxun what data you want by clicking elements in the recorder, describing it in plain English with AI Mode, starting from a pre-built auto robot, or writing code with the API or SDK. Maxun runs a real browser, handles JavaScript, pagination, infinite scroll and logins, and returns clean data. Every robot can run on demand or on a schedule, with results delivered by webhook, export or integration."}}, {"@type": "Question", "name": "What is a robot?", "acceptedAnswer": {"@type": "Answer", "text": "A robot is a reusable automation that collects web data for you. There are several types: Scrape turns a page into Markdown, HTML, text, links or screenshots; Crawl covers an entire website; Extract pulls structured rows such as products or listings; Search finds and optionally scrapes pages for a query; and Document robots extract data from or convert files such as PDFs. Every robot has adjustable inputs, like the URL, that you can change each time it runs. See Robots Overview to pick the right one."}}, {"@type": "Question", "name": "Which robot type should I use?", "acceptedAnswer": {"@type": "Answer", "text": "Use Scrape for the content of a single page, Crawl for many pages or a whole site, Extract for structured rows like prices or listings, Search to find pages for a query, and Document robots for PDFs and other files. Add Monitoring to any Scrape, Crawl or Extract robot to track changes."}}, {"@type": "Question", "name": "Is Maxun available on the cloud?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun Cloud is the fully managed version, available at app.maxun.dev. There's nothing to install or maintain."}}, {"@type": "Question", "name": "Is Maxun open source?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun is open source under the AGPLv3 license. The code is on GitHub, where you can view, modify, self-host and contribute to it."}}, {"@type": "Question", "name": "What is the difference between Maxun Cloud and the open-source version?", "acceptedAnswer": {"@type": "Answer", "text": "Both include Scrape, Crawl, Extract, Search, Monitoring, the API, SDKs, webhooks and extraction behind login. Maxun Cloud adds managed anti-bot infrastructure with CAPTCHA bypass and automatic proxy rotation, Deep Extraction, automatic adaptation to website changes, auto robots, advanced AI features, unlimited workspaces, email notifications, SAML SSO and priority support, with no servers to maintain. See the full Cloud vs Open Source comparison."}}, {"@type": "Question", "name": "How do I get started?", "acceptedAnswer": {"@type": "Answer", "text": "Sign up at app.maxun.dev/register and start your 7-day free trial. Create your first robot from a pre-built auto robot, by clicking on the data you want with the recorder, by describing it in plain English with AI Mode, or with the API and SDKs. Then run it on demand or schedule it. Most new users set up their first robot in about 5 minutes. See the Quickstart."}}, {"@type": "Question", "name": "Can I try Maxun for free?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun Cloud comes with a 7-day free trial with full access to every feature, and no credit card is required. Use it to build robots, test the API and run real extractions on your own target websites before choosing a plan. You can also self-host the open-source edition for free on your own infrastructure."}}, {"@type": "Question", "name": "How much does Maxun cost?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun Cloud has four plans, priced in USD: Starter is $19/month for 3,500 credits and 1 user with basic support. Growth is $99/month for 100,000 credits and 5 users with standard support. Pro is $399/month for 350,000 credits and 10 users with priority support. Premium has custom pricing, credits and users, with priority support and an SLA. A 7-day free trial is available with no credit card required."}}, {"@type": "Question", "name": "Which plan is right for me?", "acceptedAnswer": {"@type": "Answer", "text": "Starter is for individuals and small projects. Growth is for teams running regular extractions and monitoring. Pro is for high-volume pipelines and production workloads. Premium offers custom credits, users and data retention, an SLA, a dedicated account manager, custom integrations and fully managed onboarding."}}, {"@type": "Question", "name": "What is included in every plan?", "acceptedAnswer": {"@type": "Answer", "text": "Every plan includes Scrape, Crawl, Extract, Search and Monitoring, API access, SDKs, CLI and MCP, webhooks, scheduled runs, extraction behind login, stealth mode, and CSV and JSON exports. Higher plans add more credits, more team members, advanced stealth mode and faster support."}}, {"@type": "Question", "name": "What is a credit?", "acceptedAnswer": {"@type": "Answer", "text": "Credits are how usage is measured on Maxun Cloud. Each plan includes a set number of credits per billing cycle. Scrape: 1 credit per page. Scrape with Smart Queries: +2 credits per run. Crawl: 1 credit per page. Extract (Recorder Mode): 1 credit per 4 rows. Extract (AI Mode): 1 credit per 3 rows. Screenshot: 1 credit per screenshot. Search: 1 credit per 5 results. Search + Scrape: 1 credit per result. For example, extracting 50 products in Recorder Mode uses 12.5 credits, scraping 100 pages to Markdown uses 100 credits, and monitoring 50 pages daily with a scrape robot uses about 1,500 credits per month."}}, {"@type": "Question", "name": "Do unused credits roll over?", "acceptedAnswer": {"@type": "Answer", "text": "Unused credits usually do not carry over to the next billing cycle. However, if you upgrade your plan before your current cycle ends, your unused credits are added to your new plan's credits and remain available for the duration of your upgraded plan."}}, {"@type": "Question", "name": "Am I charged for failed requests?", "acceptedAnswer": {"@type": "Answer", "text": "No. You only pay for successful requests. If a request fails, no credits are deducted."}}, {"@type": "Question", "name": "How does billing work?", "acceptedAnswer": {"@type": "Answer", "text": "Plans are billed monthly or yearly, and credits reset at the start of each billing cycle. All prices are in USD."}}, {"@type": "Question", "name": "Do you offer refunds?", "acceptedAnswer": {"@type": "Answer", "text": "We do not offer refunds for subscription fees. The 7-day free trial lets you test all features before committing to a paid plan. If you have any issues or concerns about your subscription, contact support@maxun.dev."}}, {"@type": "Question", "name": "Do you offer custom or enterprise plans?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. The Premium plan includes custom credit and user limits, custom data retention, advanced stealth mode, fully managed onboarding and setup, priority support with an SLA, a dedicated account manager and custom integrations. For large or complex projects, our team can also run fully managed data collection for you. Contact sales for a quote."}}, {"@type": "Question", "name": "Is self-hosting free?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. The open-source edition is free under the AGPLv3 license. You only pay for your own infrastructure. See Self-Host Maxun."}}, {"@type": "Question", "name": "What sites does Maxun work on?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun is designed to work on any website. There are billions of websites out there (and hundreds are created every day). We do our best to adapt to almost every possible website - that being said there are always unique scenarios that arise, often due to inaccessible code or non-standard practices on certain sites."}}, {"@type": "Question", "name": "Can I extract structured data from any website?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. You can extract prices, contact details, listings, job posts, reviews, product specs and almost anything else visible on a page. Point and click on the data with Recorder Mode, or describe it in natural language with AI Mode."}}, {"@type": "Question", "name": "Does Maxun work on JavaScript-heavy and dynamic websites?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun runs a real browser, so it fully renders single-page apps built with React, Vue, Angular and other frameworks. It also handles infinite scroll, load more buttons, pagination, clicks and form inputs."}}, {"@type": "Question", "name": "My robot needs pagination and scrolling. Can Maxun handle this?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun supports several pagination methods to accommodate different website designs: Click on \\"next\\" to navigate to the next page: This method involves clicking a button or link that clearly indicates the next page, such as a \\"Next\\" button or an arrow pointing to the right. Click on \\"load more\\" to load more items: This method involves clicking a button that loads more items onto the current page without navigating to a separate page. Scroll down to load more items: This method involves scrolling down the page to trigger the loading of more items. This is common on websites with infinite scrolling. Scroll up to load more items: Similar to scrolling down, this method involves scrolling up to load more items. This is less common but can be found on some websites. No more items to load: This option indicates that there are no more items to load on the current page or in the entire list."}}, {"@type": "Question", "name": "Can my robot fill out a form or perform an action before extracting data?", "acceptedAnswer": {"@type": "Answer", "text": "Definitely! Your robot can: Open a webpage. Log in. Click on buttons. Fill out a form. Select from a dropdown menu, radios, checkboxes, dates, times, etc. Extract structured data from a webpage into a spreadsheet. Take screenshots."}}, {"@type": "Question", "name": "Can my robot log in to websites?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. On Maxun Cloud, the Maxun Chrome extension securely reuses your existing logged-in browser session, so you never share your username or password with Maxun. Self-hosted robots can use encrypted, locally stored credentials. See Extract Behind Login."}}, {"@type": "Question", "name": "Can I extract data from an iFrame?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. From Maxun 0.0.6 onwards, data inside iFrames can be extracted."}}, {"@type": "Question", "name": "Can I extract data from Shadow DOM?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. From Maxun 0.0.6 onwards, data inside Shadow DOM can be extracted."}}, {"@type": "Question", "name": "Can I extract data from a list page and each item's detail page?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Deep Extraction captures a list and then visits each item's detail page to collect more data, all in one robot. It's available on Maxun Cloud."}}, {"@type": "Question", "name": "Can I reuse a robot on similar pages?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Duplicate a robot with a new URL to extract the same data from other pages with the same structure, without training a new robot. This lets you bulk-extract data from thousands of pages."}}, {"@type": "Question", "name": "What happens when a website changes its layout?", "acceptedAnswer": {"@type": "Answer", "text": "On Maxun Cloud, robots automatically adapt to most website layout and structure changes, so your data keeps flowing without rewriting anything. On the open-source edition, you can retrain a robot with point and click in minutes."}}, {"@type": "Question", "name": "What formats can Maxun return data in?", "acceptedAnswer": {"@type": "Answer", "text": "Scrape and Crawl robots return clean Markdown optimized for LLMs, as well as HTML, plain text, links, AI summaries and screenshots. Extract robots return structured rows you can get as JSON through the API or webhooks, export as CSV or JSON, or send to Google Sheets and Airtable."}}, {"@type": "Question", "name": "Can Maxun extract data from PDFs and documents?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Document robots extract specific fields from, or convert, PDF, CSV, XLSX, DOCX, JPG and PNG files up to 10 MB into Markdown, HTML, links or a summary. Document robots are in beta."}}, {"@type": "Question", "name": "Can I download files using Maxun?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun is primarily designed to extract text data, but this is a part of our roadmap. Currently, you can extract URLs of files, such as: Image URLs: When training a robot, you can capture image URLs by clicking on the image and selecting \\"URL.\\" This gives you a list of image URLs that you can download manually or with other tools. File links: Similarly, you can capture the URLs of other types of files, such as PDFs or documents, by selecting the link or element that points to the file."}}, {"@type": "Question", "name": "Can I monitor websites for changes?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Schedule any robot to run hourly, daily or on a custom interval and turn on Monitoring. Each run is compared with the previous one so you can see exactly what changed in prices, stock, listings or content. On Maxun Cloud, you can also get email alerts."}}, {"@type": "Question", "name": "Can I use Maxun for large-scale data collection?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun can extract from thousands of pages at once and supports long-running tasks, making it suitable for large datasets, extensive market research and high-volume pipelines. Plans scale up to 350,000 credits per month, and Premium plans offer custom limits. For complex needs, our team also runs fully managed data collection."}}, {"@type": "Question", "name": "Can I use Maxun if I don't write code?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Alongside the API and SDKs, Maxun has a no-code dashboard. Build a robot in minutes by clicking on the data you want, describing it in plain English, or starting from a pre-built auto robot, then schedule it and export the results."}}, {"@type": "Question", "name": "What APIs and SDKs are available?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun provides a REST API, official SDKs for Node.js/TypeScript and Python, a CLI for terminal workflows, an MCP server for AI agents, and webhooks for real-time delivery."}}, {"@type": "Question", "name": "How do I authenticate API requests?", "acceptedAnswer": {"@type": "Answer", "text": "Generate an API key in the Maxun dashboard and send it in the x-api-key header. On Maxun Cloud, the base URL is https://app.maxun.dev and all endpoints live under /api, for example GET /api/robots. See API Key & Authentication."}}, {"@type": "Question", "name": "Does Maxun work with AI agents and MCP?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Maxun has an MCP server, so agents in Claude, Cursor, Windsurf, Cline and other MCP-compatible tools can run your robots and get results through natural language. See MCP Setup and MCP Tools. There is also a Claude Code skill."}}, {"@type": "Question", "name": "Which integrations does Maxun support?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun sends data to Google Sheets, Airtable, n8n and any URL via webhooks. For AI apps, it works with LangChain, LangGraph, LlamaIndex, Mastra, the OpenAI SDK, the Vercel AI SDK, Claude Code and OpenClaw."}}, {"@type": "Question", "name": "How do I get results without polling the API?", "acceptedAnswer": {"@type": "Answer", "text": "Use webhooks. Maxun sends a POST request with the run's data to your URL as soon as a run finishes, and also when a run fails."}}, {"@type": "Question", "name": "What do I need to self-host Maxun?", "acceptedAnswer": {"@type": "Answer", "text": "The quickest way is Docker Compose, which runs everything for you. For a local development setup you need Node.js 18+, PostgreSQL and MinIO. For production with your own domain and HTTPS, see the production self-hosting guide."}}, {"@type": "Question", "name": "Is web scraping legal?", "acceptedAnswer": {"@type": "Answer", "text": "Collecting publicly available data is legal in many jurisdictions, but it depends on the data, the website's terms and how you use it. We recommend reviewing each website's terms of service, avoiding personal data without a lawful basis under laws such as GDPR, and scraping at a reasonable rate. This is not legal advice; consult a lawyer for your specific use case."}}, {"@type": "Question", "name": "How does Maxun ensure ethical data collection?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun is built for responsible use. We encourage collecting publicly available information, checking each website's terms of service, and using rate limiting so extractions don't overwhelm websites."}}, {"@type": "Question", "name": "Does Maxun store my passwords?", "acceptedAnswer": {"@type": "Answer", "text": "No, not on Maxun Cloud. The password-free login flow uses the Maxun Chrome extension to sync your existing browser session, so no passwords are shared or stored. On self-hosted installations, credentials entered while training a robot are encrypted and stored locally on your own instance. See Extract Behind Login."}}, {"@type": "Question", "name": "Can my account be flagged when scraping behind a login?", "acceptedAnswer": {"@type": "Answer", "text": "Possibly. IP address changes and automated logins can trigger security checks on some websites. Use Maxun Cloud with the Chrome extension for authenticated extraction, avoid automated logins on sensitive websites, and never use critical personal accounts for scraping."}}, {"@type": "Question", "name": "Can Maxun solve CAPTCHAs?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun Cloud solves several types of CAPTCHA (such as reCAPTCHA and hCaptcha), but not custom CAPTCHAs. CAPTCHA bypass is not supported in the open-source edition."}}, {"@type": "Question", "name": "What about websites running A/B tests?", "acceptedAnswer": {"@type": "Answer", "text": "If a website is running an A/B test and the robot encounters a different version of the page than the one it was trained on, it might either fail or collect incorrect information. While the robot can adapt to certain differences, it may not handle all variations effectively."}}, {"@type": "Question", "name": "What about websites with very strong bot detection?", "acceptedAnswer": {"@type": "Answer", "text": "Maxun Cloud bypasses bot detection effectively with stealth mode and managed proxies. For the open-source edition, bring your own proxies. However, if your robot needs to log into a website, there is a higher chance of runs failing because: The same user is accessing the account from different IP addresses (your local IP and Maxun's IPs). High run frequency by the robot can appear suspicious. As a result, robots that require login credentials are more likely to be flagged. Running login robots less frequently can reduce the flagging rate. See Stealth for more ways to avoid blocks."}}, {"@type": "Question", "name": "I have more questions!", "acceptedAnswer": {"@type": "Answer", "text": "We're here to help! Write to us at support@maxun.dev, or open an issue on GitHub."}}]}`}
  </script>
</head>

# Frequently Asked Questions

Answers to common questions about Maxun: what it is, pricing and credits, robots, the API and SDKs, self-hosting, security and known limitations. Can't find your answer? Email support@maxun.dev.

**Jump to:** [General](#general) · [Pricing & Billing](#pricing--billing) · [Robots & Data Extraction](#robots--data-extraction) · [API, SDK & AI Agents](#api-sdk--ai-agents) · [Legal & Security](#legal--security) · [Limitations](#limitations) · [Support](#support)

## General

### What is Maxun?
Maxun is an open-source web data platform that turns any website into clean, structured data for your apps, workflows and AI agents. You can scrape, crawl, extract, search and monitor the web through the [REST API](/api/api), [Node.js and Python SDKs](/sdks), [CLI](/category/cli) and [MCP server](/mcp/setup), or through a no-code dashboard where anyone can build robots by clicking on a page or describing what they need in plain English.

### Who is Maxun for?
Maxun is built for both developers and non-technical teams. Developers and AI engineers use the API, SDKs, CLI and MCP server to power products and agents. Growth, operations and research teams use the no-code dashboard to build and schedule robots without writing code. Both work on the same platform, so teams can hand work back and forth.

### What can I build with Maxun?
Teams use Maxun for [price and competitor monitoring](/usecases/ecommerce_automation), [lead generation](/usecases/lead_generation), [market research](/usecases/market_research), RAG pipelines, AI agents that need live web data, [real estate](/usecases/real_estate) and job aggregators, review tracking and [content aggregation](/usecases/content_aggregation). If the data is on a website, you can turn it into a dataset, a scheduled feed or an API.

### How does Maxun work?
You tell Maxun what data you want by clicking elements in the recorder, describing it in plain English with AI Mode, starting from a pre-built auto robot, or writing code with the API or SDK. Maxun runs a real browser, handles JavaScript, pagination, infinite scroll and logins, and returns clean data. Every robot can run on demand or on a [schedule](/robot-schedule), with results delivered by [webhook](/api/webhooks), export or [integration](/category/integrations).

### What is a robot?
A robot is a reusable automation that collects web data for you. There are several types: **Scrape** turns a page into Markdown, HTML, text, links or screenshots; **Crawl** covers an entire website; **Extract** pulls structured rows such as products or listings; **Search** finds and optionally scrapes pages for a query; and **Document** robots extract data from or convert files such as PDFs. Every robot has adjustable inputs, like the URL, that you can change each time it runs. See [Robots Overview](/robot/robots) to pick the right one.

### Which robot type should I use?
Use [Scrape](/robot/scrape) for the content of a single page, [Crawl](/robot/crawl) for many pages or a whole site, [Extract](/category/extract) for structured rows like prices or listings, [Search](/robot/search/search-introduction) to find pages for a query, and [Document](/robot/document) robots for PDFs and other files. Add [Monitoring](/monitoring) to any Scrape, Crawl or Extract robot to track changes.

### Is Maxun available on the cloud?
Yes. Maxun Cloud is the fully managed version, available at [app.maxun.dev](https://app.maxun.dev). There's nothing to install or maintain.

### Is Maxun open source?
Yes. Maxun is open source under the AGPLv3 license. The code is on [GitHub](https://github.com/getmaxun/maxun), where you can view, modify, self-host and contribute to it.

### What is the difference between Maxun Cloud and the open-source version?
Both include Scrape, Crawl, Extract, Search, Monitoring, the API, SDKs, webhooks and extraction behind login. Maxun Cloud adds managed anti-bot infrastructure with CAPTCHA bypass and automatic proxy rotation, [Deep Extraction](/deep-extraction), automatic adaptation to website changes, auto robots, advanced AI features, unlimited workspaces, email notifications, SAML SSO and priority support, with no servers to maintain. See the full [Cloud vs Open Source](/cloud-vs-oss) comparison.

### How do I get started?
Sign up at [app.maxun.dev/register](https://app.maxun.dev/register) and start your 7-day free trial. Create your first robot from a pre-built auto robot, by clicking on the data you want with the recorder, by describing it in plain English with AI Mode, or with the API and SDKs. Then run it on demand or schedule it. Most new users set up their first robot in about 5 minutes. See the [Quickstart](/quickstart).

## Pricing & Billing

### Can I try Maxun for free?
Yes. Maxun Cloud comes with a 7-day free trial with full access to every feature, and no credit card is required. Use it to build robots, test the API and run real extractions on your own target websites before choosing a plan. You can also [self-host](/category/self-host) the open-source edition for free on your own infrastructure.

### How much does Maxun cost?
Maxun Cloud has four plans. All prices are in USD.

| Plan | Price | Credits per month | Users | Support |
|---|---|---|---|---|
| Starter | $19/month | 3,500 | 1 | Basic |
| Growth | $99/month | 100,000 | 5 | Standard |
| Pro | $399/month | 350,000 | 10 | Priority |
| Premium | Custom | Custom | Custom | Priority with SLA |

See [maxun.dev/pricing](https://maxun.dev/pricing) for the latest plans.

### Which plan is right for me?
Starter is for individuals and small projects. Growth is for teams running regular extractions and monitoring. Pro is for high-volume pipelines and production workloads. Premium offers custom credits, users and data retention, an SLA, a dedicated account manager, custom integrations and fully managed onboarding.

### What is included in every plan?
Every plan includes Scrape, Crawl, Extract, Search and Monitoring, API access, SDKs, CLI and MCP, webhooks, scheduled runs, extraction behind login, stealth mode, and CSV and JSON exports. Higher plans add more credits, more team members, advanced stealth mode and faster support.

### What is a credit?
Credits are how usage is measured on Maxun Cloud. Each plan includes a set number of credits per billing cycle.

| Operation | Cost |
|---|---|
| Scrape | 1 credit per page |
| Scrape with Smart Queries | +2 credits per run |
| Crawl | 1 credit per page |
| Extract (Recorder Mode) | 1 credit per 4 rows |
| Extract (AI Mode) | 1 credit per 3 rows |
| Screenshot | 1 credit per screenshot |
| Search | 1 credit per 5 results |
| Search + Scrape | 1 credit per result |

For example, extracting 50 products in Recorder Mode uses 12.5 credits, scraping 100 pages to Markdown uses 100 credits, and monitoring 50 pages daily with a scrape robot uses about 1,500 credits per month.

### Do unused credits roll over?
Unused credits usually do not carry over to the next billing cycle. However, if you upgrade your plan before your current cycle ends, your unused credits are added to your new plan's credits and remain available for the duration of your upgraded plan.

### Am I charged for failed requests?
No. You only pay for successful requests. If a request fails, no credits are deducted.

### How does billing work?
Plans are billed monthly or yearly, and credits reset at the start of each billing cycle. All prices are in USD.

### Do you offer refunds?
We do not offer refunds for subscription fees. The 7-day free trial lets you test all features before committing to a paid plan. If you have any issues or concerns about your subscription, contact support@maxun.dev.

### Do you offer custom or enterprise plans?
Yes. The Premium plan includes custom credit and user limits, custom data retention, advanced stealth mode, fully managed onboarding and setup, priority support with an SLA, a dedicated account manager and custom integrations. For large or complex projects, our team can also run fully managed data collection for you. [Contact sales](https://maxun.dev/talk-to-sales) for a quote.

### Is self-hosting free?
Yes. The open-source edition is free under the AGPLv3 license. You only pay for your own infrastructure. See [Self-Host Maxun](/category/self-host).

## Robots & Data Extraction

### What sites does Maxun work on?
Maxun is designed to work on any website. There are billions of websites out there (and hundreds are created every day). We do our best to adapt to almost every possible website - that being said there are always unique scenarios that arise, often due to inaccessible code or non-standard practices on certain sites.

### Can I extract structured data from any website?
Yes. You can extract prices, contact details, listings, job posts, reviews, product specs and almost anything else visible on a page. Point and click on the data with [Recorder Mode](/robot/extract/robot-actions), or describe it in natural language with [AI Mode](/robot/extract/llm-extraction).

### Does Maxun work on JavaScript-heavy and dynamic websites?
Yes. Maxun runs a real browser, so it fully renders single-page apps built with React, Vue, Angular and other frameworks. It also handles infinite scroll, load more buttons, pagination, clicks and form inputs.

### My robot needs pagination and scrolling. Can Maxun handle this?
Yes. Maxun supports several pagination methods to accommodate different website designs:
1. **Click on "next" to navigate to the next page**: This method involves clicking a button or link that clearly indicates the next page, such as a "Next" button or an arrow pointing to the right.
2. **Click on "load more" to load more items**: This method involves clicking a button that loads more items onto the current page without navigating to a separate page.
3. **Scroll down to load more items**: This method involves scrolling down the page to trigger the loading of more items. This is common on websites with infinite scrolling.
4. **Scroll up to load more items**: Similar to scrolling down, this method involves scrolling up to load more items. This is less common but can be found on some websites.
5. **No more items to load**: This option indicates that there are no more items to load on the current page or in the entire list.

### Can my robot fill out a form or perform an action before extracting data?
Definitely! Your robot can:
- Open a webpage
- Log in
- Click on buttons
- Fill out a form
- Select from a dropdown menu, radios, checkboxes, dates, times, etc.
- Extract structured data from a webpage into a spreadsheet
- Take screenshots

### Can my robot log in to websites?
Yes. On Maxun Cloud, the Maxun Chrome extension securely reuses your existing logged-in browser session, so you never share your username or password with Maxun. Self-hosted robots can use encrypted, locally stored credentials. See [Extract Behind Login](/extract-login).

### Can I extract data from an iFrame?
Yes. From Maxun 0.0.6 onwards, data inside iFrames can be extracted.

### Can I extract data from Shadow DOM?
Yes. From Maxun 0.0.6 onwards, data inside Shadow DOM can be extracted.

### Can I extract data from a list page and each item's detail page?
Yes. [Deep Extraction](/deep-extraction) captures a list and then visits each item's detail page to collect more data, all in one robot. It's available on Maxun Cloud.

### Can I reuse a robot on similar pages?
Yes. [Duplicate a robot](/robot/extract/robot-duplicate) with a new URL to extract the same data from other pages with the same structure, without training a new robot. This lets you bulk-extract data from thousands of pages.

### What happens when a website changes its layout?
On Maxun Cloud, robots automatically adapt to most website layout and structure changes, so your data keeps flowing without rewriting anything. On the open-source edition, you can [retrain a robot](/robot/extract/robot-retrain) with point and click in minutes.

### What formats can Maxun return data in?
Scrape and Crawl robots return clean Markdown optimized for LLMs, as well as HTML, plain text, links, AI summaries and screenshots. Extract robots return structured rows you can get as JSON through the API or webhooks, export as CSV or JSON, or send to [Google Sheets](/integrations/gsheet) and [Airtable](/integrations/airtable).

### Can Maxun extract data from PDFs and documents?
Yes. [Document robots](/robot/document) extract specific fields from, or convert, PDF, CSV, XLSX, DOCX, JPG and PNG files up to 10 MB into Markdown, HTML, links or a summary. Document robots are in beta.

### Can I download files using Maxun?
Maxun is primarily designed to extract text data, but this is a part of our roadmap.
Currently, you can extract `URLs` of files, such as:

1. **Image URLs**: When training a robot, you can capture image URLs by clicking on the image and selecting "URL." This gives you a list of image URLs that you can download manually or with other tools.
2. **File links**: Similarly, you can capture the URLs of other types of files, such as PDFs or documents, by selecting the link or element that points to the file.

### Can I monitor websites for changes?
Yes. [Schedule](/robot-schedule) any robot to run hourly, daily or on a custom interval and turn on [Monitoring](/monitoring). Each run is compared with the previous one so you can see exactly what changed in prices, stock, listings or content. On Maxun Cloud, you can also get email alerts.

### Can I use Maxun for large-scale data collection?
Yes. Maxun can extract from thousands of pages at once and supports long-running tasks, making it suitable for large datasets, extensive market research and high-volume pipelines. Plans scale up to 350,000 credits per month, and Premium plans offer custom limits. For complex needs, our team also runs fully managed data collection.

### Can I use Maxun if I don't write code?
Yes. Alongside the API and SDKs, Maxun has a no-code dashboard. Build a robot in minutes by clicking on the data you want, describing it in plain English, or starting from a pre-built auto robot, then schedule it and export the results.

## API, SDK & AI Agents

### What APIs and SDKs are available?
Maxun provides a [REST API](/api/api), official SDKs for [Node.js/TypeScript](/sdk/node-sdk/sdk-overview) and [Python](/sdk/python-sdk/sdk-overview), a [CLI](/cli/cli-overview) for terminal workflows, an [MCP server](/mcp/setup) for AI agents, and [webhooks](/api/webhooks) for real-time delivery.

### How do I authenticate API requests?
Generate an API key in the Maxun dashboard and send it in the `x-api-key` header. On Maxun Cloud, the base URL is `https://app.maxun.dev/api/v1`, for example `GET /api/v1/robots`. See [API Key & Authentication](/api/api).

### Does Maxun work with AI agents and MCP?
Yes. Maxun has an MCP server, so agents in Claude, Cursor, Windsurf, Cline and other MCP-compatible tools can run your robots and get results through natural language. See [MCP Setup](/mcp/setup) and [MCP Tools](/mcp/tools). There is also a [Claude Code skill](/integrations/claude-code).

### Which integrations does Maxun support?
Maxun sends data to [Google Sheets](/integrations/gsheet), [Airtable](/integrations/airtable), [n8n](/integrations/n8n) and any URL via [webhooks](/api/webhooks). For AI apps, it works with [LangChain](/integrations/langchain), [LangGraph](/integrations/langgraph), [LlamaIndex](/integrations/llamaindex), [Mastra](/integrations/mastra), the [OpenAI SDK](/integrations/openai), the [Vercel AI SDK](/integrations/vercel-ai-sdk), [Claude Code](/integrations/claude-code) and [OpenClaw](/integrations/openclaw).

### How do I get results without polling the API?
Use [webhooks](/api/webhooks). Maxun sends a POST request with the run's data to your URL as soon as a run finishes, and also when a run fails.

### What do I need to self-host Maxun?
The quickest way is [Docker Compose](/installation/docker), which runs everything for you. For a [local development setup](/installation/local) you need Node.js 18+, PostgreSQL and MinIO. For production with your own domain and HTTPS, see the [production self-hosting guide](/self-host).

## Legal & Security

### Is web scraping legal?
Collecting publicly available data is legal in many jurisdictions, but it depends on the data, the website's terms and how you use it. We recommend reviewing each website's terms of service, avoiding personal data without a lawful basis under laws such as GDPR, and scraping at a reasonable rate. This is not legal advice; consult a lawyer for your specific use case.

### How does Maxun ensure ethical data collection?
Maxun is built for responsible use. We encourage collecting publicly available information, checking each website's terms of service, and using rate limiting so extractions don't overwhelm websites.

### Does Maxun store my passwords?
No, not on Maxun Cloud. The password-free login flow uses the Maxun Chrome extension to sync your existing browser session, so no passwords are shared or stored. On self-hosted installations, credentials entered while training a robot are encrypted and stored locally on your own instance. See [Extract Behind Login](/extract-login).

### Can my account be flagged when scraping behind a login?
Possibly. IP address changes and automated logins can trigger security checks on some websites. Use Maxun Cloud with the Chrome extension for authenticated extraction, avoid automated logins on sensitive websites, and never use critical personal accounts for scraping.

## Limitations

### Can Maxun solve CAPTCHAs?
Maxun Cloud solves several types of CAPTCHA (such as reCAPTCHA and hCaptcha), but not custom CAPTCHAs. CAPTCHA bypass is not supported in the open-source edition.

### What about websites running A/B tests?
If a website is running an A/B test and the robot encounters a different version of the page than the one it was trained on, it might either fail or collect incorrect information. While the robot can adapt to certain differences, it may not handle all variations effectively.

### What about websites with very strong bot detection?
Maxun Cloud bypasses bot detection effectively with stealth mode and managed proxies. For the open-source edition, [bring your own proxies](/byop).
However, if your robot needs to log into a website, there is a higher chance of runs failing because:
1. The same user is accessing the account from different IP addresses (your local IP and Maxun's IPs).
2. High run frequency by the robot can appear suspicious.

As a result, robots that require login credentials are more likely to be flagged. Running login robots less frequently can reduce the flagging rate. See [Stealth](/stealth) for more ways to avoid blocks.

## Support

### I have more questions!
We're here to help! Write to us at support@maxun.dev, or open an issue on [GitHub](https://github.com/getmaxun/maxun/issues).

</page>

<page url="https://docs.maxun.dev/development/i18n" title="Translate Maxun (i18n)">

# i18n Translation

The Maxun interface is available in 9 languages: English, Spanish, German, Japanese, Chinese, Korean, Turkish, Hebrew and Azerbaijani. Follow this guide to add a new language or improve an existing one. We love community contributions!

## Add a New Language
1. Navigate to the <a href="https://github.com/getmaxun/maxun/tree/develop/public/locales">public/locales</a> directory.

2. Create your own language file (e.g., `es.json` for Spanish) in the <a href="https://github.com/getmaxun/maxun/tree/develop/public/locales">public/locales</a> directory.

3. Update the content of the file to include translations for your platform.

4. Open a pull request to the `develop` branch.

## Edit an Existing Language File

1. In the <a href="https://github.com/getmaxun/maxun/tree/develop/public/locales">public/locales</a> directory, make changes to the language file you want to update (e.g., `en.json` for English).

2. Open a pull request to the `develop` branch.

</page>

<page url="https://docs.maxun.dev/development/repo" title="Maxun GitHub Repositories & Code Structure">

# Repository Structure

Our main app, SDKs and documentation are separate repositories under the <a href="https://github.com/getmaxun/">Maxun organization</a> on GitHub.

## Maxun
- Maxun repository: <a href="https://github.com/getmaxun/maxun">https://github.com/getmaxun/maxun</a>
- `src`: Maxun frontend. We use React, Typescript and Styled Components.
- `server`: Maxun backend. We use Node.js, Typescript, Playwright.
- `maxun-core`: Maxun core. Available on <a href="https://www.npmjs.com/package/maxun-core">NPM</a>.
- `public/locales`: All i18n language files.

## Maxun Docs
Maxun Docs Repository: <a href="https://github.com/getmaxun/maxun_docs">https://github.com/getmaxun/maxun_docs</a>

## Maxun SDKs
- Maxun Node.js SDK: <a href="https://github.com/getmaxun/node-sdk">https://github.com/getmaxun/node-sdk</a>
- Maxun Python SDK:  <a href="https://github.com/getmaxun/python-sdk">https://github.com/getmaxun/python-sdk</a>

</page>

<page url="https://docs.maxun.dev/development/contributing" title="Contributing to Maxun">

# Contributing to Maxun

Maxun is open source and we welcome contributions, from bug fixes and new features to docs and translations. Here's how to get your first pull request merged.

## Local Setup
Read local installation instructions here: <a href="/installation/local">Local Installation</a>

## Open Your First Pull Request

### 1. Create a new branch

Create a new branch from the develop branch. Use a descriptive name for your branch, such as `feature/new-feature` or `fix/bug-fix`. This makes it easier to understand the purpose of the changes.


### 2. Make your changes

Make the necessary code changes and commit them to your local branch.

Follow Conventional Commits: Use the format `<type>(<scope>): <short description>[optional body][optional footer]` for your commit messages.

Example: `feat(api): Add new endpoint for user data`

### 3. Push your changes
Push your local branch to the remote repository:
```git push origin <your-branch-name>```


### 4. Create a Pull Request

1. Go to the repository on GitHub and navigate to the `Pull Requests` tab.
2. Click `New pull request`.
3. Select your branch as the "compare" branch and develop as the "base" branch.
4. Give your PR a descriptive title that summarizes the changes.
5. Write a clear and concise description of the changes you made and why they are necessary.
6. Add any relevant screenshots or GIFs to help visualize the changes.


### 5. Review and Merge

1. Once you submit the PR, it will be reviewed by other developers.
2. Address any comments or requested changes from the reviewers.
3. Once the PR is approved, it will be merged into the develop branch.

### 6. Remember

1. Always test your changes thoroughly before submitting a PR.
2. Keep your PRs focused on a single feature or bug fix.
3. Be respectful and responsive to feedback from reviewers.

### 7. AI-Assisted Contributions

AI-assisted contributions are welcome. If you use AI tools to generate code, please ensure that:

1. The changes fully address the issue or feature request.
2. The code is tested and works as expected.
3. The implementation follows the existing project structure and conventions.
4. You understand the code you are submitting and can respond to review feedback.

Low-quality, unverified, or blindly generated patches will not be merged.

</page>