Before you start
An active proxy connection with its assigned protocol, hostname, port, full username and password. Copy those fields from your account or activation details. A browser profile does not create proxy credentials.
Python 3.10 or newer and Scrapy 2.20 or newer. This recipe uses an HTTP proxy endpoint for HTTP and HTTPS destination pages. TLS-to-proxy and SOCKS connections require a compatible download handler and are outside this example.
A proxy for the downloader
Scrapy sends requests through its downloader middleware. HttpProxyMiddleware reads the proxy URL from request metadata and handles HTTP proxy authentication.
- Proxy every downloader request
- A small middleware assigns the proxy to the page, robots.txt and callback-generated requests.
- Encoded credentials
- URL-encode the username and password before building an authenticated proxy URL.
- Normal spider output
- Keep selectors, item fields and export formats in your spider.
- Verified HTTPS responses
- The example enables server certificate verification and bounds concurrency, download time and retries.
1. Install Scrapy in a virtual environment
Keep the spider dependencies separate from your system Python. Activate the environment before installing or running the spider.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'scrapy>=2.20'2. Set your connection details
Use an http://HOST:PORT proxy server URL for this recipe, even when TARGET_URL uses https://. Supply the raw username and password; the middleware URL-encodes them when constructing the proxy URL.
export PROXY_SERVER='http://YOUR_PROXY_HOST:YOUR_PORT'
export PROXY_USERNAME='YOUR_PROXY_USERNAME'
export PROXY_PASSWORD='YOUR_PROXY_PASSWORD'
export TARGET_URL='https://example.com/'3. Run the connection-check spider
Save scrapy_proxy.py and run it without creating a full Scrapy project. The spider respects robots.txt and exports the page URL, status and title.
The spider explicitly enables DOWNLOAD_VERIFY_CERTIFICATES. Keep certificate verification enabled for HTTPS destinations.
scrapy runspider scrapy_proxy.py -O results.jsonimport os
from urllib.parse import quote, urlsplit, urlunsplit
import scrapy
class CrawlhubProxyMiddleware:
def __init__(self):
server = urlsplit(os.environ['PROXY_SERVER'])
if server.scheme != 'http' or not server.hostname or server.username or server.password or server.query or server.fragment or server.path not in ('', '/'):
raise ValueError('Use an HTTP proxy server URL without credentials, a path or a query.')
user = quote(os.environ['PROXY_USERNAME'], safe='')
password = quote(os.environ['PROXY_PASSWORD'], safe='')
self.proxy = urlunsplit((server.scheme, f'{user}:{password}@{server.netloc}', '', '', ''))
def process_request(self, request):
# Include robots.txt and requests generated by callbacks.
request.meta['proxy'] = self.proxy
class ProxyCheckSpider(scrapy.Spider):
name = 'crawlhub_proxy_check'
custom_settings = {
'DOWNLOADER_MIDDLEWARES': {CrawlhubProxyMiddleware: 740},
'CONCURRENT_REQUESTS': 1,
'DOWNLOAD_TIMEOUT': 30,
'DOWNLOAD_VERIFY_CERTIFICATES': True,
'RETRY_TIMES': 1,
'ROBOTSTXT_OBEY': True,
}
async def start(self):
target = os.environ.get('TARGET_URL', 'https://example.com/')
yield scrapy.Request(target, callback=self.parse)
def parse(self, response):
yield {
'url': response.url,
'status': response.status,
'title': response.css('title::text').get(),
}4. Expand the spider carefully
Inspect results.json before adding pagination or more URLs. The supplied downloader middleware applies the proxy consistently, including robots.txt. Keep that middleware enabled when moving the spider into a larger project.
A successful request confirms that the page loaded, but it does not by itself prove the exit IP. Run the same code against an IP-check endpoint you trust, inspect the returned address, then switch to your target. Keep the requested location and session settings consistent when comparing results.
If a destination disallows the request in robots.txt, use a different allowed test destination instead of treating that result as a proxy failure.
Troubleshooting
407 from the downloader
Confirm the proxy URL and credential encoding. Review sanitized logs without exposing the authenticated proxy URL.
Some requests bypass the proxy
Keep the proxy middleware enabled before HttpProxyMiddleware. A proxy set only on the first page request can leave robots.txt or later requests on a direct route.
No items exported
Check robots.txt decisions, HTTP statuses and selectors. Successful transport does not guarantee a matching title element.
