How to use Crawlera with selenium (Python, Chrome, Windows) without Polipo

Question

So basically i am trying to use the Crawlera Proxy from scrapinghub with selenium chrome on windows using python.

I checked the documentation and they suggested using Polipo like this:

1) adding the following lines to /etc/polipo/config

parentProxy = "proxy.crawlera.com:8010" parentAuthCredentials = "<CRAWLERA_APIKEY>:" 

2) adding this to selenium driver

polipo_proxy = "127.0.0.1:8123" proxy = Proxy({ 'proxyType': ProxyType.MANUAL, 'httpProxy': polipo_proxy, 'ftpProxy' : polipo_proxy, 'sslProxy' : polipo_proxy, 'noProxy' : '' }) capabilities = dict(DesiredCapabilities.CHROME) proxy.add_to_capabilities(capabilities) driver = webdriver.Chrome(desired_capabilities=capabilities) 

Now i'd like to not use Polipo and directly use the proxy.

Is there a way to replace the polipo_proxy variable and change it to the crawlera one? Each time i try to do it, it doesn't take it into account and runs without proxy.

Crawlera proxy format is like the folowwing: [API KEY]:@[HOST]:[PORT]

I tried adding the proxy using the following line:

chrome_options.add_argument('--proxy-server=http://[API KEY]:@[HOST]:[PORT]) 

but the problem is that i need to specify HTTP and HTTPS differently.

Thank you in advance!

Answer 1

Polipo is no longer maintained and hence there are challenges in using it. Crawlera requires Authentication, which Chrome driver does not seem to support as of now. You can try using Firefox webdriver, in that you can set the proxy authentication in the custom Firefox profile and use the profile as shown in Running selenium behind a proxy server and http://toolsqa.com/selenium-webdriver/http-proxy-authentication/.

I have been suffering from the same problem and got some relief out of it. Hope it will help you as well. To solve this problem you have to use Firefox driver and its profile to put proxy information this way.

profile = webdriver.FirefoxProfile() profile.set_preference("network.proxy.type", 1) profile.set_preference("network.proxy.http", "proxy.server.address") profile.set_preference("network.proxy.http_port", "port_number") profile.update_preferences() driver = webdriver.Firefox(firefox_profile=profile) 

This totally worked for me. For reference you can use above sites.

Answer 2

Scrapinghub creates a new project. You need to set up a forwarding agent by using apikey, and then set webdriver to use this agent. The project address is: zyte-smartproxy-headless-proxy

You can have a look