Node.js | Crawlhub Developers

Prepare the client

Install Undici with npm install undici. Set PROXY_SERVER, PROXY_USERNAME and PROXY_PASSWORD in your private runtime environment.

Connection example

Node.js proxy example

import { ProxyAgent, fetch } from 'undici';

const { PROXY_SERVER, PROXY_USERNAME, PROXY_PASSWORD } = process.env;
if (!PROXY_SERVER || !PROXY_USERNAME || !PROXY_PASSWORD) {
  throw new Error('Set PROXY_SERVER, PROXY_USERNAME and PROXY_PASSWORD.');
}
const agent = new ProxyAgent({
  uri: PROXY_SERVER, // http://host:port, without credentials
  token: `Basic ${Buffer.from(`${PROXY_USERNAME}:${PROXY_PASSWORD}`).toString('base64')}`,
});
try {
  const response = await fetch('https://example.com/', {
    dispatcher: agent,
    signal: AbortSignal.timeout(30000),
    redirect: 'manual',
  });
  console.log(response.status);
  console.log((await response.text()).slice(0, 500));
} finally {
  await agent.close();
}

Verify the connection

Start with a public test destination you are allowed to access. Confirm authentication, exit IP and timeouts before adding target-specific parsing. Never disable TLS verification to make a failing connection appear successful.

When it fails

Review Troubleshooting for authentication, DNS, connection and TLS errors. Include your client version and a sanitized error when contacting support.