I have written a Python script for web scraping using Selenium with Firefox (Geckodriver).
The script runs perfectly on Windows, but when I run it on Linux — either as a Python script or packaged as an executable — it fails with the following error:
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=58083): Read timed out. (read timeout=120) Full traceback attached (see below).
I am already using the latest
geckodriver.I have increased the timeout settings.
I run my own server.
I have set Firefox to run in headless mode on both Windows and Linux.
Despite all this, the issue only happens on Linux.
Does anyone know why this could happen and how I can fix it?
Any help would be really appreciated!
Error message:
Exception in thread Thread-1 (scrap_amazon): Traceback (most recent call last): File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 292, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/socket.py", line 707, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner self.run() File "/usr/lib/python3.12/threading.py", line 1010, in run self._target(*self._args, **self._kwargs) File "/var/nova/sourcecode/Python-Scrap-Tool/scraper_amazon/_init_.py", line 239, in scrap_amazon driver = create_driver() ^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/scraper_amazon/_init_.py", line 210, in create_driver driver = webdriver.Firefox(service=service, options=options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/selenium/webdriver/firefox/webdriver.py", line 71, in _init_ super()._init_(command_executor=executor, options=options) File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 250, in _init_ self.start_session(capabilities) File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 342, in start_session response = self.execute(Command.NEW_SESSION, caps)["value"] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 427, in execute response = self.command_executor.execute(driver_command, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/selenium/webdriver/remote/remote_connection.py", line 404, in execute return self._request(command_info[0], url, body=data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/selenium/webdriver/remote/remote_connection.py", line 428, in _request response = self._conn.request(method, url, body=body, headers=headers, timeout=self._client_config.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/_request_methods.py", line 143, in request return self.request_encode_body( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body return self.urlopen(method, url, **extra_kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/poolmanager.py", line 443, in urlopen response = conn.urlopen(method, u.request_uri, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/var/nova/sourcecode/Python-Scrap-Tool/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=58083): Read timed out. (read timeout=120) 
