How to grab headers in python selenium-webdriver

Question

I am trying to grab the headers in selenium webdriver. Something similar to the following:

>>> import requests >>> res=requests.get('http://google.com') >>> print res.headers 

I need to use the Chrome webdriver because it supports flash and some other things that I need to test a web page. Here is what I have so far in Selenium:

from selenium import webdriver driver = webdriver.Chrome() driver.get('https://login.comcast.net/login?r=comcast.net&s=oauth&continue=https%3A%2F%2Flogin.comcast.net%2Foauth%2Fauthorize%3Fclient_id%3Dxtv-account-selector%26redirect_uri%3Dhttps%3A%2F%2Fxtv-pil.xfinity.com%2Fxtv-authn%2Fxfinity-cb%26response_type%3Dcode%26scope%3Dopenid%2520https%3A%2F%2Flogin.comcast.net%2Fapi%2Flogin%26state%3Dhttps%3A%2F%2Ftv.xfinity.com%2Fpartner-success.html%26prompt%3Dlogin%26response%3D1&reqId=18737431-624b-44cb-adf0-2a85d91bd662&forceAuthn=1&client_id=xtv-account-selector') driver.find_element_by_css_selector('#user').send_keys('[email protected]') driver.find_element_by_css_selector('#passwd').send_keys('XXY') driver.find_element_by_css_selector('#passwd').submit() print driver.headers ### How to do this? 

I have seen some other answers that recommend running an entire selenium server to get this information (https://github.com/derekargueta/selenium-profiler). How would I get it using something similar to the above with Webdriver?

Answer 1

Unfortunately, you cannot get this information from the Selenium webdriver, nor will you be able to any time in the near future it seems. An excerpt from a very long conversation on the subject:

This feature isn't going to happen.

The gist of the main reason being, from what I gather from the discussion, that the webdriver is meant for "driving the browser", and extending the API beyond that primary goal will, in the opinion of the developers, cause the overall quality and reliability of the API to suffer.

One potential workaround that I have seen suggested in a number of places, including the conversation linked above, is to use BrowserMob Proxy, which can be used to capture HTTP content, and can be used with selenium - though the linked example does not use the Python selenium API. It does seem that there is a Python wrapper for BrowserMob Proxy, but I cannot vouch for it's efficacy since I have never used it.

Answer 2

Now, it is very easy i suppose https://pypi.org/project/selenium-wire/ it is an extension of selenium. use from seleniumwire import webdriver and proceed as usual.

Answer 3

You could try Mobilenium, a python package (still in development) that binds BrowserMob Proxy and Selenium.

An usage example:

>>> from mobilenium import mobidriver >>> >>> browsermob_path = 'path/to/browsermob-proxy' >>> mob = mobidriver.Firefox(browsermob_binary=browsermob_path) >>> mob.get('http://python-requests.org') 301 >>> mob.response['redirectURL'] 'http://docs.python-requests.org' >>> mob.headers['Content-Type'] 'application/json; charset=utf8' >>> mob.title 'Requests: HTTP for Humans \u2014 Requests 2.13.0 documentation' >>> mob.find_elements_by_tag_name('strong')[1].text 'Behold, the power of Requests' 

Answer 4

You can get the header via the log (source from Mma's answer)

from selenium import webdriver import json driver = webdriver.PhantomJS(executable_path=r"your_path") har = json.loads(driver.get_log('har')[0]['message']) # get the log print('headers: ', har['log']['entries'][0]['request']['headers']) 

Answer 5

You can use the JAVASCRIPT built-in method.

It only can be done once the driver has already been created though.

from selenium import webdriver driver = webdriver.Chrome() # Store it in a variable and print the value agent = driver.execute_script("return navigator.userAgent") print(agent) # directly print the value print(driver.execute_script("return navigator.userAgent"))