setting request headers in selenium

Question

I'm attempting to set the request header 'Referer' to spoof a request coming from another site. We need the ability test that a specific referrer is used, which returns a specific form to the user, otherwise an alternative form is given.

I can do this within poltergeist by:

page.driver.headers = {"Referer" => referer_string} 

but I can't find the equivalent functionality for the selemium driver.

How can I set request headers in the capybara selenium driver?

Answer 1

For those people using Python, you may consider using Selenium Wire which can set request headers as well as provide you with the ability to inspect requests and responses.

from seleniumwire import webdriver # Import from seleniumwire # Create a new instance of the Chrome driver (or Firefox) driver = webdriver.Chrome() # Create a request interceptor def interceptor(request): del request.headers['Referer'] # Delete the header first request.headers['Referer'] = 'some_referer' # Set the interceptor on the driver driver.request_interceptor = interceptor # All requests will now use 'some_referer' for the referer driver.get('https://mysite') 

Install with:

pip install selenium-wire 

Answer 2

Webdriver doesn't contain an API to do it. See issue 141 from Selenium tracker for more info. The title of the issue says that it's about response headers but it was decided that Selenium won't contain API for request headers in scope of this issue. Several issues about adding API to set request headers have been marked as duplicates: first, second, third.

Here are a couple of possibilities that I can propose:

1. Use another driver/library instead of selenium
2. Write a browser-specific plugin (or find an existing one) that allows you to add header for request.
3. Use browsermob-proxy or some other proxy.

I'd go with option 3 in most of cases. It's not hard.

Note that Ghostdriver has an API for it but it's not supported by other drivers.

Answer 3

I had the same issue. I solved it downloading modify-headers firefox add-on and activate it with selenium.

The code in python is the following

fp = webdriver.FirefoxProfile() path_modify_header = 'C:/xxxxxxx/modify_headers-0.7.1.1-fx.xpi' fp.add_extension(path_modify_header) fp.set_preference("modifyheaders.headers.count", 1) fp.set_preference("modifyheaders.headers.action0", "Add") fp.set_preference("modifyheaders.headers.name0", "Name_of_header") # Set here the name of the header fp.set_preference("modifyheaders.headers.value0", "value_of_header") # Set here the value of the header fp.set_preference("modifyheaders.headers.enabled0", True) fp.set_preference("modifyheaders.config.active", True) fp.set_preference("modifyheaders.config.alwaysOn", True) driver = webdriver.Firefox(firefox_profile=fp) 

Answer 4

Had the same issue today, except that I needed to set different referer per test. I ended up using a middleware and a class to pass headers to it. Thought I'd share (or maybe there's a cleaner solution?):

lib/request_headers.rb: class CustomHeadersHelper cattr_accessor :headers end class RequestHeaders def initialize(app, helper = nil) @app, @helper = app, helper end def call(env) if @helper headers = @helper.headers if headers.is_a?(Hash) headers.each do |k,v| env["HTTP_#{k.upcase.gsub("-", "_")}"] = v end end end @app.call(env) end end 

---

config/initializers/middleware.rb require 'request_headers' if %w(test cucumber).include?(Rails.env) Rails.application.config.middleware.insert_before Rack::Lock, "RequestHeaders", CustomHeadersHelper end 

---

spec/support/capybara_headers.rb require 'request_headers' module CapybaraHeaderHelpers shared_context "navigating within the site" do before(:each) { add_headers("Referer" => Capybara.app_host + "/") } end def add_headers(custom_headers) if Capybara.current_driver == :rack_test custom_headers.each do |name, value| page.driver.browser.header(name, value) end else CustomHeadersHelper.headers = custom_headers end end end 

---

spec/spec_helper.rb ... config.include CapybaraHeaderHelpers 

Then I can include the shared context wherever I need, or pass different headers in another before block. I haven't tested it with anything other than Selenium and RackTest, but it should be transparent, as header injection is done before the request actually hits the application.

Answer 5

I wanted something a bit slimmer for RSpec/Ruby so that the custom code only had to live in one place. Here's my solution:

/spec/support/selenium.rb ... RSpec.configure do |config| config.after(:suite) do $custom_headers = nil end end module RequestWithExtraHeaders def headers $custom_headers.each do |key, value| self.set_header "HTTP_#{key}", value end if $custom_headers super end end class ActionDispatch::Request prepend RequestWithExtraHeaders end 

Then in my specs:

/specs/features/something_spec.rb ... $custom_headers = {"Referer" => referer_string} 

Answer 6

If you are using javacsript and only want to implement on chrome, Puppeteer is the best option as it has native support to modify headers. Check this out: https://pptr.dev/#?product=Puppeteer&version=v10.1.0&show=api-pagesetextrahttpheadersheaders

Although for cross-browser usage you might check out @requestly/selenium npm package. It is a wrapper around requestly extension to enable easy integration in selenium-webdriver.The extension can modify headers. Check out: https://www.npmjs.com/package/@requestly/selenium

Answer 7

Setting request headers in the web driver directly does not work. This is true.

However, you can work around this problem by using the browser devtools (I tested with edge & chrome) and this works perfectly.

According to the documentation, you have the possibility to add custom headers: https://chromedevtools.github.io/devtools-protocol/tot/Network/

Please find below an example.

 [Test] public async Task AuthenticatedRequest() { await LogMessage("=== starting the test ==="); EdgeOptions options = new EdgeOptions {UseChromium = true}; options.AddArgument("no-sandbox"); var driver = new RemoteWebDriver(new Uri(_testsSettings.GridUrl), options.ToCapabilities(), TimeSpan.FromMinutes(3)); //Get DevTools IDevTools devTools = driver; //DevTools Session var session = devTools.GetDevToolsSession(); var devToolsSession = session.GetVersionSpecificDomains<DevToolsSessionDomains>(); await devToolsSession.Network.Enable(new Network.EnableCommandSettings()); var extraHeader = new Network.Headers(); var data = await Base64KerberosTicket(); var headerValue = $"Negotiate {data}"; await LogMessage($"header values is {headerValue}"); extraHeader.Add("Authorization", headerValue); await devToolsSession.Network.SetExtraHTTPHeaders(new Network.SetExtraHTTPHeadersCommandSettings { Headers = extraHeader }); driver.Url = _testsSettings.TestUrl; driver.Navigate(); driver.Quit(); await LogMessage("=== ending the test ==="); } 

This is an example written in C# but the same shall probably work with java, python as well as the major platforms.

Hope it helps the community.

Answer 8

In c# selenium, an option is to use IWebDriver and add network interceptors that appropriately update the headers.

driver.Manage().Network.AddRequestHandler(new NetworkRequestHandler() { RequestMatcher = (u) => true, RequestTransformer = (u) => { u.Headers["my-header"] = "some-value"; return u; } }); 

Answer 9

If you use the HtmlUnitDriver, you can set request headers by modifying the WebClient, like so:

final case class Header(name: String, value: String) final class HtmlUnitDriverWithHeaders(headers: Seq[Header]) extends HtmlUnitDriver { super.modifyWebClient { val client = super.getWebClient headers.foreach(h => client.addRequestHeader(h.name, h.value)) client } } 

The headers will then be on all requests made by the web browser.

Answer 10

With the solutions already discussed above the most reliable one is using Browsermob-Proxy

But while working with the remote grid machine, Browsermob-proxy isn't really helpful.

This is how I fixed the problem in my case. Hopefully, might be helpful for anyone with a similar setup.

1. Add the ModHeader extension to the chrome browser

How to download the Modheader? Link

ChromeOptions options = new ChromeOptions(); options.addExtensions(new File(C://Downloads//modheader//modheader.crx)); // Set the Desired capabilities DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability(ChromeOptions.CAPABILITY, options); // Instantiate the chrome driver with capabilities WebDriver driver = new RemoteWebDriver(new URL(YOUR_HUB_URL), options); 

2. Go to the browser extensions and capture the Local Storage context ID of the ModHeader

3. Navigate to the URL of the ModHeader to set the Local Storage Context

.

// set the context on the extension so the localStorage can be accessed driver.get("chrome-extension://idgpnmonknjnojddfkpgkljpfnnfcklj/_generated_background_page.html"); Where `idgpnmonknjnojddfkpgkljpfnnfcklj` is the value captured from the Step# 2 

4. Now add the headers to the request using Javascript

.

 ((Javascript)driver).executeScript( "localStorage.setItem('profiles', JSON.stringify([{ title: 'Selenium', hideComment: true, appendMode: '', headers: [ {enabled: true, name: 'token-1', value: 'value-1', comment: ''}, {enabled: true, name: 'token-2', value: 'value-2', comment: ''} ], respHeaders: [], filters: [] }]));"); 

Where token-1, value-1, token-2, value-2 are the request headers and values that are to be added.

5. Now navigate to the required web-application.

   driver.get("your-desired-website");

Answer 11

You can do it with PhantomJSDriver.

PhantomJSDriver pd = ((PhantomJSDriver) ((WebDriverFacade) getDriver()).getProxiedDriver()); pd.executePhantomJS( "this.onResourceRequested = function(request, net) {" + " net.setHeader('header-name', 'header-value')" + "};"); 

Using the request object, you can filter also so the header won't be set for every request.

Answer 12

If you just need to set the User-Agent header, there is an option for Chrome:

chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"') 

Now the browser sends User-Agent.