Configuring proxies on Selenium + Python3 + Chrome WebDriver + Docker

Question

I can't connect using proxies using Selenium Chrome WebDriver on Python 3.7

1 - When using a proxy (adding -proxy-server=%s on chrome_options)

def selenium_connect(): PROXY = "66.97.38.58:80" url = "http://whatsmyip.org" chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') chrome_options.add_argument('--proxy-server=%s' % PROXY) driver = webdriver.Chrome(chrome_options=chrome_options) driver.get(url) 

In this case I get an empty result.

2 - I have tried this alternative: How do you run headless chrome and a proxy using selenium in python?

def selenium_connect(): PROXY = "66.97.38.58:80" url = "http://whatsmyip.org" chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') desired_caps = chrome_options.to_capabilities() prox = Proxy() prox.proxy_type = ProxyType.MANUAL prox.http_proxy = PROXY prox.add_to_capabilities(desired_caps) driver = webdriver.Chrome(chrome_options=chrome_options, desired_capabilities=desired_caps) driver.get(url) 

I this case, the connection is routed via my local ip and not the proxy's ip. I've uploaded the function in docker here:

https://github.com/gerimo/challenge

I'm using a standard ubuntu Docker environment, selenium-3.141.0, chromium-chromedriver 68.0

Answer 1

This is not the shortest of solutions, but I thought it might be beneficial for someone. I managed to use a remote webdriver connected to the selenium hub with a luminati proxy in a docker container:

from selenium import webdriver from zipfile import ZipFile PROXY_HOST = 'servercountry-gb.zproxy.luminati.io' PROXY_PORT = 22225 PROXY_USER = 'lum-customer-####-zone-####-country-gb-dns-remote' PROXY_PASS = '#####' chrome_options = webdriver.ChromeOptions() manifest_json = """ { "version": "1.0.0", "manifest_version": 2, "name": "Chrome Proxy", "permissions": [ "proxy", "tabs", "unlimitedStorage", "storage", "<all_urls>", "webRequest", "webRequestBlocking" ], "background": { "scripts": ["background.js"] }, "minimum_chrome_version":"50.0.0" } """ background_js = """ var config = { mode: "fixed_servers", rules: { singleProxy: { scheme: "http", host: "%s", port: parseInt(%s) }, bypassList: ["localhost"] } }; chrome.proxy.settings.set({value: config, scope: "regular"}, function() {}); function callbackFn(details) { return { authCredentials: { username: "%s", password: "%s" } }; } chrome.webRequest.onAuthRequired.addListener( callbackFn, {urls: ["<all_urls>"]}, ['blocking'] ); """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS) pluginfile = 'proxy_auth_plugin.zip' with ZipFile(pluginfile, 'w') as zp: zp.writestr('manifest.json', manifest_json) zp.writestr('background.js', background_js) chrome_options.add_extension(pluginfile) 

Once we have authorised ourselves we could then use this somewhere in our code:

with webdriver.Remote(options= chrome_options, command_executor = 'http://selenium-hub:4444/wd/hub') as driver: driver.get('http://ipconfig.me') # whatever you need to do 

I have encountered a strange issue where I cannot use headless chrome but non-headless chrome seemed to be fine. The remote webdriver was headless in the docker container. (python 3.7.4)