MCP

Before you start

An active Crawlhub proxy and its assigned host, port, username and password. PROXY_SERVER includes the protocol and port, but no username or password. Use the values from your account; these examples do not activate a proxy.

Python 3.10 or newer and an MCP client that can launch a local stdio server. This example targets the official Python SDK 1.x API, pinned below major version 2.

This is a custom server example, not a hosted Crawlhub MCP endpoint. It fetches response text without running JavaScript and does not provide a built-in search tool.

What this MCP integration does

Model Context Protocol connects an AI client to tools. This guide builds your own local server with the official Python SDK. The proxy handles the outbound request; the MCP client calls the tool and reads its result.

A tool you can inspect: The downloadable Python server defines one tool: fetch_configured_page.
A configured destination: The server operator sets TARGET_URL. The tool accepts no arbitrary destination argument.
Bounded page text: The example returns up to 20,000 response bytes, along with the source URL, status and content type.
Local stdio transport: Connect the process to an MCP client that supports local stdio servers.

1. Install the SDK and HTTP client

Create a virtual environment and install the SDK 1.x line with HTTPX. Save mcp_server.py from the next step in the same project folder.

Run in your terminal

python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'mcp>=1.30,<2' 'httpx>=0.28,<1'

2. Configure the server

Set the proxy variables and TARGET_URL in the environment used by the server process. In a desktop client, supply them in the client’s private server configuration or launch environment, rather than assuming it inherits your interactive shell.

Run in your terminal

export PROXY_SERVER='http://YOUR_PROXY_HOST:YOUR_PORT'
export PROXY_USERNAME='YOUR_PROXY_USERNAME'
export PROXY_PASSWORD='YOUR_PROXY_PASSWORD'
export TARGET_URL='https://example.com/'

Download mcp_server.py

mcp_server.py

import os
from urllib.parse import quote, urlsplit, urlunsplit
import httpx
from mcp.server.fastmcp import FastMCP

server = urlsplit(os.environ['PROXY_SERVER'])
if server.scheme not in ('http', 'https') or not server.hostname or server.username or server.password or server.query or server.fragment or server.path not in ('', '/'):
    raise ValueError('Use an HTTP(S) proxy server URL without credentials, a path or a query.')
username = quote(os.environ['PROXY_USERNAME'], safe='')
password = quote(os.environ['PROXY_PASSWORD'], safe='')
proxy_url = urlunsplit((server.scheme, f'{username}:{password}@{server.netloc}', '', '', ''))
target = os.environ.get('TARGET_URL', 'https://example.com/')
if urlsplit(target).scheme not in ('http', 'https'):
    raise ValueError('TARGET_URL must be an HTTP(S) page.')

mcp = FastMCP('Crawlhub page reader')

@mcp.tool()
def fetch_configured_page() -> dict:
    """Fetch the public page configured by the server operator."""
    with httpx.Client(proxy=proxy_url, timeout=30, trust_env=False,
                      follow_redirects=False) as client:
        with client.stream('GET', target) as response:
            response.raise_for_status()
            # Keep model context and download size bounded.
            parts = bytearray()
            limit = 20000
            truncated = False
            for chunk in response.iter_bytes():
                remaining = limit - len(parts)
                parts.extend(chunk[:remaining])
                if len(chunk) > remaining:
                    truncated = True
                    break
            return {
                'url': str(response.url),
                'status': response.status_code,
                'content_type': response.headers.get('content-type', ''),
                'body': parts.decode(response.encoding or 'utf-8', errors='replace'),
                'truncated': truncated,
            }

if __name__ == '__main__':
    mcp.run(transport='stdio')

3. Connect a stdio-capable MCP client

Create a local server entry in your client. Set its executable to the absolute path of .venv/bin/python and its argument to the absolute path of mcp_server.py. Select stdio transport where the client asks.

Start or restart the server from the client, inspect its tool list, and confirm that fetch_configured_page appears. A server started in an ordinary terminal waits for MCP messages; it is not an HTTP website.

Client setting	Example to replace
Command	/absolute/path/to/project/.venv/bin/python
Argument	/absolute/path/to/project/mcp_server.py
Transport	stdio
Environment	PROXY_SERVER, PROXY_USERNAME, PROXY_PASSWORD, TARGET_URL

4. Call the tool and check its output

Ask the client to call fetch_configured_page. Inspect url, status, content_type, body and truncated. The example does not follow redirects automatically: set TARGET_URL to the final allowed page URL if the destination redirects.

The tool returns raw page text, which may be HTML. Use a parser or a browser-based integration when the task needs extracted fields or JavaScript-rendered content. Keep the returned URL alongside any answer that uses the page.

To test the proxy route, configure an IP-check URL first. Restart the server after changing environment variables, then call the tool again.

Troubleshooting

Tool does not appear

Check the absolute executable path, installed SDK version and client stderr logs. Keep stdout reserved for MCP messages.

407 or connection error

Verify the proxy environment in the server process, not only in your terminal.

Unexpected page text

Check the content type, URL and status. HTML, redirects and JavaScript-only pages need different handling.

Next steps

Browse integration guides or review the connection fields.

Open the tool documentation