Before you start
An active Crawlhub proxy and its assigned host, port, username and password. PROXY_SERVER includes the protocol and port, but no username or password. Use the values from your account; these examples do not activate a proxy.
Python with Selenium 4, Node.js supported by proxy-chain, and compatible Chrome/ChromeDriver on the same machine. This guide uses a local HTTP bridge with an HTTP(S) upstream proxy.
Browser routing and authentication
Selenium sets the browser’s proxy route. A small local bridge handles the upstream username and password, so your browser does not need an authentication extension or a handler for every website challenge.
- Local browser endpoint
- The bridge listens on 127.0.0.1 and forwards to your assigned Crawlhub proxy.
- Upstream authentication
- The Node.js bridge supplies the proxy login. Selenium only receives the local address.
- Standard Selenium code
- Keep your existing Chrome options, locators and waits.
- Two processes, clear lifetime
- Keep the bridge running while the browser works. Stop it when the job finishes.
1. Install Selenium and Chrome
Create a Python environment and install Selenium. Install Chrome and a compatible driver before the connection check. Selenium Manager can obtain the driver when network access is available.
Install proxy-chain in the same project folder as selenium_bridge.mjs. The bridge and browser should run on the same machine.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install selenium
npm init -y
npm install proxy-chain2. Start the authenticated bridge
Save both helper files below in the same project folder. The CLI wrapper imports proxy_bridge.mjs.
Set your assigned proxy values in the first terminal, then save and run selenium_bridge.mjs. It prints a local URL such as http://127.0.0.1:49152. The port is selected automatically.
Leave that terminal running. The bridge binds only to loopback and prints no upstream credentials. Stop it with Ctrl+C when you are done.
export PROXY_SERVER='http://YOUR_PROXY_HOST:YOUR_PORT'
export PROXY_USERNAME='YOUR_PROXY_USERNAME'
export PROXY_PASSWORD='YOUR_PROXY_PASSWORD'
node selenium_bridge.mjsimport { Server } from 'proxy-chain';
export async function createProxyBridge() {
const { PROXY_SERVER, PROXY_USERNAME, PROXY_PASSWORD } = process.env;
if (!PROXY_SERVER || !PROXY_USERNAME || !PROXY_PASSWORD) {
throw new Error('Set PROXY_SERVER, PROXY_USERNAME and PROXY_PASSWORD.');
}
const upstream = new URL(PROXY_SERVER);
if (!['http:', 'https:'].includes(upstream.protocol) ||
upstream.username || upstream.password || upstream.search ||
upstream.hash || (upstream.pathname && upstream.pathname !== '/')) {
throw new Error('Use an HTTP(S) proxy server URL without credentials, a path or a query.');
}
upstream.username = PROXY_USERNAME;
upstream.password = PROXY_PASSWORD;
const server = new Server({
host: '127.0.0.1',
port: 0,
prepareRequestFunction: () => ({ upstreamProxyUrl: upstream.href }),
});
await server.listen();
return {
url: `http://127.0.0.1:${server.port}`,
close: () => server.close(true),
};
}import { createProxyBridge } from './proxy_bridge.mjs';
const bridge = await createProxyBridge();
console.log(bridge.url);
for (const signal of ['SIGINT', 'SIGTERM']) {
process.once(signal, async () => { await bridge.close(); process.exit(0); });
}3. Run Selenium through the local URL
Open a second terminal, activate the Python environment, and save selenium_proxy.py in the project folder. Set LOCAL_PROXY_SERVER to the exact URL printed by the bridge, not the example port shown here.
The script routes both HTTP and HTTPS browser traffic through the bridge, sets a page-load timeout and closes the driver after printing the title.
source .venv/bin/activate
export LOCAL_PROXY_SERVER='http://127.0.0.1:PORT_PRINTED_BY_BRIDGE'
export TARGET_URL='https://example.com/'
python selenium_proxy.pyimport os
from urllib.parse import urlsplit
from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType
server = urlsplit(os.environ['LOCAL_PROXY_SERVER'])
if server.scheme != 'http' or server.hostname != '127.0.0.1' or not server.port:
raise ValueError('Set LOCAL_PROXY_SERVER to the URL printed by the local bridge.')
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = server.netloc
proxy.ssl_proxy = server.netloc
options = webdriver.ChromeOptions()
options.proxy = proxy
options.add_argument('--headless=new')
driver = webdriver.Chrome(options=options)
try:
driver.set_page_load_timeout(30)
driver.get(os.environ.get('TARGET_URL', 'https://example.com/'))
print(driver.title)
finally:
driver.quit()4. Verify the session
A successful request confirms that the page loaded, but it does not by itself prove the exit IP. Run the same code against an IP-check endpoint you trust, inspect the returned address, then switch to your target. Keep the requested location and session settings consistent when comparing results.
Keep the browser and assigned sticky proxy session together for a related workflow. Use Selenium waits for the specific content you need instead of assuming that page navigation means all asynchronous content is ready.
Troubleshooting
Connection refused at localhost
Keep the bridge running and copy its printed port exactly. Restarting the bridge may choose a new port.
Upstream 407
Check the three proxy environment variables in the bridge terminal, then restart the bridge after correcting them.
Driver cannot start
Resolve Chrome and driver installation first. For Selenium Grid or a remote browser, the bridge must be reachable from the browser host; this loopback example is for a local browser.
