Maxun + MCP: Give Your AI Agent Scraping Robots

Say you ask an agent to check three competitors' pricing pages every Monday. It works, technically. It opens a browser, takes screenshots, scrolls, sometimes clicks "Monthly" instead of "Annual", and comes back with a slightly different table each week. Dozens of tool calls and a pile of image tokens to read three prices.

Nothing about that job needs intelligence on every run. Where the prices sit on the page doesn't change from week to week. What you want is to show a tool once where the data is, then let the agent call that tool and get the same clean rows every time. That's what Maxun does, and it ships an MCP server. Here's how to wire it up, including the part of the setup that surprised me.

Robots Instead of Browsing

Maxun is an open-source (AGPL-3.0) web data platform. You open a page in its recorder, click the things you want, and it saves a robot: a replayable extraction that returns structured rows. There's also an AI mode where you describe the data in plain English, and on self-hosted installs it works with Anthropic, OpenAI or Ollama.

For an agent, the trade-off is simple. A robot is fast, cheap in tokens and deterministic. It's also brittle: when the site changes its layout, the robot returns empty lists until you retrain it. I'll take that over a model that is creatively wrong every week.

What you need:

• Maxun on Elestio. It starts at $16/month on a 2 CPU / 4 GB instance, which is enough for a handful of robots. Heavy crawls need more RAM because every run drives a real Chromium.
• Python 3.10+ on the machine that runs your agent.
• An MCP client: Claude Code, Claude Desktop, Cursor or anything else that speaks MCP.

Step 1: Record a Robot

In the Maxun dashboard, create an Extract robot and enter the page URL. In the recorder:

1. Use Capture List, click one item (a pricing card, a product row), then click the fields inside it.
2. Tell it how pagination works: Click "Next", Click "Load more", Scroll down, or No more items if everything is on one page.
3. Set a limit on the number of items, and save the robot with a name you'll recognize later.

Use Capture Text for single values like a headline price. Run the robot once from the dashboard and check the output before an agent ever sees it.

Step 2: Get an API Key and the Robot ID

Under Settings > API Key, generate a key. Then list your robots:

curl -s https://your-maxun-domain/api/robots \
  -H "x-api-key: YOUR_API_KEY" | python3 -m json.tool

Each item has an id. Copy the one for your robot.

The Part That Surprised Me

Look closely at that response. In the open-source API, the key identifies a user, but the robot endpoints don't filter by that user. I read the source (main branch, October 18): GET /api/robots returns every robot on the instance, and running a robot or reading its runs only needs a valid key and a robot ID. There's also a duplicate endpoint that copies a robot onto any target URL you pass it.

So a Maxun key isn't scoped to one agent. Whoever holds any key can run any robot, read every run's data, and point copies of your robots at new URLs from your server's IP. The practical rule: one Maxun instance is one trust zone. Don't mix the robots that touch your logged-in admin pages with the ones you hand to an experimental agent. Give risky work its own instance.

That's also why I didn't use the bundled MCP worker. It's a stdio server you build from the repo with npm run mcp:build, it passes through whatever the API returns (so list_robots shows everything), and in the build I read, run_robot checks .length on the extracted data, which the API returns as objects. The result can come back with run metadata and no rows.

Step 3: An 85-Line MCP Server With an Allowlist

mkdir -p ~/maxun-mcp && cd ~/maxun-mcp
python3 -m venv .venv
.venv/bin/pip install "mcp>=2.3,<3" httpx

Save this as ~/maxun-mcp/maxun_mcp.py and put your robot ID in ALLOWED. The description is what the model reads when deciding which robot to call, so make it specific.

import os

import httpx
from mcp.server.mcpserver import MCPServer

MAXUN_URL = os.environ["MAXUN_URL"].rstrip("/")
API_KEY = os.environ["MAXUN_API_KEY"]
MAX_ROWS = int(os.environ.get("MAX_ROWS", "200"))

# The only robots this agent may run: robot ID -> what it returns.
ALLOWED = {
    "replace-with-robot-id": "Competitor pricing page: plan name, monthly price, seats included",
}

client = httpx.Client(
    base_url=f"{MAXUN_URL}/api",
    headers={"x-api-key": API_KEY},
    timeout=httpx.Timeout(10.0, read=300.0),
)
mcp = MCPServer("maxun-robots")


def _denied(robot_id: str) -> dict | None:
    if robot_id not in ALLOWED:
        return {"status": "denied", "hint": "Not an allowed robot. Call list_robots first."}
    return None


def _trim(captured):
    # Captured data is a list of rows, groups of lists, or a single row.
    if isinstance(captured, list):
        return captured[:MAX_ROWS]
    if isinstance(captured, dict):
        return {k: v[:MAX_ROWS] if isinstance(v, list) else v for k, v in captured.items()}
    return captured


def _result(run: dict) -> dict:
    data = run.get("data") or {}
    return {
        "run_id": run.get("runId"),
        "status": run.get("status"),
        "finished_at": run.get("finishedAt"),
        "changed_since_last_run": run.get("hasChanges", False),
        "text": _trim(data.get("textData") or {}),
        "lists": _trim(data.get("listData") or {}),
    }


@mcp.tool()
def list_robots() -> dict[str, str]:
    """List the scraping robots you may run and what each one returns."""
    return ALLOWED


@mcp.tool()
def run_robot(robot_id: str) -> dict:
    """Run an allowed robot now and return the data it extracted. Can take minutes."""
    if denied := _denied(robot_id):
        return denied
    try:
        resp = client.post(f"/robots/{robot_id}/runs", json={}, headers={"x-run-source": "mcp"})
    except httpx.ReadTimeout:
        return {"status": "still_running", "hint": "Do not run it again. Call latest_result in a few minutes."}
    if resp.status_code == 500:
        return {"status": "failed", "hint": "The run failed. Check the robot's run history in Maxun."}
    resp.raise_for_status()
    return _result(resp.json()["run"])


@mcp.tool()
def latest_result(robot_id: str) -> dict:
    """Return the most recent successful run of an allowed robot without running it."""
    if denied := _denied(robot_id):
        return denied
    resp = client.get(f"/robots/{robot_id}/runs")
    resp.raise_for_status()
    runs = [r for r in resp.json()["runs"]["items"] if r.get("status") == "success"]
    if not runs:
        return {"status": "no_successful_run"}
    return _result(max(runs, key=lambda r: r.get("finishedAt") or ""))


if __name__ == "__main__":
    mcp.run()

Why the 5-minute timeout matters: the Maxun run endpoint doesn't return until the run finishes, polling every two seconds for up to three hours. Without a timeout, a slow site ties up the agent's tool call until the client gives up. Without the "do not run it again" hint, the agent may retry and start a second run.

Step 4: Connect It to Your Agent

For Claude Code:

claude mcp add maxun \
  -e MAXUN_URL=https://your-maxun-domain \
  -e MAXUN_API_KEY=YOUR_API_KEY \
  -- ~/maxun-mcp/.venv/bin/python ~/maxun-mcp/maxun_mcp.py

For Claude Desktop or Cursor, add it to the mcpServers block of the config file, with absolute paths:

{
  "mcpServers": {
    "maxun": {
      "command": "/home/you/maxun-mcp/.venv/bin/python",
      "args": ["/home/you/maxun-mcp/maxun_mcp.py"],
      "env": {
        "MAXUN_URL": "https://your-maxun-domain",
        "MAXUN_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Then ask: "Run the competitor pricing robot and tell me if anything changed." It should call list_robots, then run_robot, and use changed_since_last_run instead of comparing tables by eye.

One caveat about the key: if your agent has a shell on the same machine, it can read the key from that config. Then the wrapper is a convenience and the Maxun instance is your real boundary, which is one more reason to keep a separate instance for agent work.

When the Robot Breaks

Sites change. Self-hosted Maxun doesn't adapt robots automatically (that's a Maxun Cloud feature), so you retrain the robot in the dashboard when its output goes empty. Proxy rotation and CAPTCHA solving are also Cloud-only; on your own instance you bring your own proxy.

Troubleshooting

401 "API key is missing" or 403 "Invalid API key". The x-api-key header is missing or wrong. Each user has a single key: to rotate it, delete it under Settings > API Key and generate a new one.

404 on /api/robots. MAXUN_URL points somewhere that doesn't route /api to the backend. Test with the curl from Step 2 before touching the agent.

"status": "failed" on every run. The API returns a generic 500 for any failed run. Open the robot's run history in the dashboard to see the real error, such as a timeout or a blocked page.

Runs succeed but the lists are empty. The page layout changed. Retrain the robot.

The agent runs the same robot twice. Your MCP client's tool timeout is probably shorter than five minutes, so it gave up and retried before the wrapper answered. Lower read=300.0 below your client's limit so still_running comes back first.

Wrapping Up

Record the boring part once, give the agent three tools with an allowlist, and keep one instance per trust zone. Deploy Maxun on Elestio and keep the Maxun docs open while you record your first robot. If your agent also needs to store what it collects, our guide on S3 storage for AI agents covers scoped keys and undo.

Thanks for reading ❤️ See you in the next one 👋