How to open url in browser with custom headers using python

Question

I have tried using the webbrowser module, with the code below but want to set custom headers such as:

'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1' 

Here's the code I have right now:

import webbrowser webbrowser.open("https://www.bing.com/search?q=TEST123") 

I'm okay with using another library too. Essentially I want my python script to open a url in my default browser with custom headers.

Answer 1

The documentation for the webbrowser module doesn't provide information on how to access underlying headers. It seems this is not possible. As stated in the documentation:

The webbrowser module provides a high-level interface to allow displaying Web-based documents to users.

1. Use an extension/addon on your browser

You could use your current code and install an extension on your browser like simple-modify-headers extension for Firefox or Chrome. (The extension can be installed via this link for Firefox and via this link for Chrome).

It's very easy to change header values with these extensions. For simple-modify-headers:

There are lots of other extensions/addons out there but I can't name all of them here. Just search "Modify Header extension addon [your browser]" to find one that fits your needs.

2. Use another library

You could use Selenium Wire. This library might be exactly what you want:

Selenium Wire extends Selenium's Python bindings to give you access to the underlying requests made by the browser. You author your code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses and making changes to them on the fly.

Example:

Install via pip:

pip install selenium-wire 

Download and install a driver for your browser: Chrome Driver or Gecko Driver.

Choose a version compatible with your browser.

To get your browser version: on Firefox go to menu > help > about; on Chrome go to menu > about chrome

Install OpenSSL:

# For apt based Linux systems sudo apt install openssl 

See the documentation for more details on the installation.

from seleniumwire import webdriver # Import from seleniumwire # Create a new instance of the Chrome driver (or Firefox) driver = webdriver.Chrome() # Create a request interceptor def interceptor(request): del request.headers['User-Agent'] # Delete the header first request.headers['User-Agent'] = 'Custom User-Agent' # Set the interceptor on the driver driver.request_interceptor = interceptor # All requests will now use 'some_referer' for the referer driver.get('https://www.bing.com/search?q=TEST123') 

I derived the code above from this answer.

You can see Selenium's documentation for other browsers, if you need it.