I am referring to these questions:
I used solution from second question to handle all suggested https protocols:
System.getProperties().setProperty("https.protocols", "TLSv1.2,TLSv1.1,TLSv1,SSLv3");
Full code:
/** * Opens HTTP/HTTPS and setup connection to given URL. * * @return prepared HttpURLConnection connection * @throws IOException in case URL is malformed or connection cannot be established */ public void openHttpUrlConnectionForGet() throws IOException { // Set Https protocols System.getProperties().setProperty("https.protocols", "TLSv1.2,TLSv1.1,TLSv1,SSLv3"); // Create connection URL urlObject = new URL(location); HttpURLConnection conn; if (proxy != null) { InetSocketAddress adr = new InetSocketAddress(proxy.getPk().getAddress(), proxy.getPk().getPort()); java.net.Proxy prx = new java.net.Proxy(java.net.Proxy.Type.HTTP, adr); conn = (HttpURLConnection) urlObject.openConnection(prx); } else { conn = (HttpURLConnection) urlObject.openConnection(); } conn.setRequestMethod("GET"); conn.setInstanceFollowRedirects(false); // Setup SSL factory if (conn instanceof HttpsURLConnection) { HttpsURLConnection httpsc = (HttpsURLConnection) conn; if (hostnameVerifier != null) { httpsc.setHostnameVerifier(hostnameVerifier); } if (trustManager != null) { try { SSLContext sc = SSLContext.getInstance("SSL"); sc.init(null, new TrustManager[]{trustManager}, new java.security.SecureRandom()); httpsc.setSSLSocketFactory(sc.getSocketFactory()); } catch (KeyManagementException | NoSuchAlgorithmException e) { throw new RuntimeException("Cannot init HTTPS connection.", e); } } } // Configure timeouts conn.setConnectTimeout(connectTimeout); conn.setDefaultUseCaches(false); conn.setUseCaches(false); conn.setDoOutput(false); conn.setReadTimeout(readTimeout); // Check connection if (StringUtils.isEmpty(userAgent)) { throw new IllegalArgumentException("Cannot create HTTP(S) connection. User-Agent header is not set."); } if (StringUtils.isEmpty(accept)) { throw new IllegalArgumentException("Cannot create HTTP(S) connection. Accept header is not set."); } if (StringUtils.isEmpty(acceptLanguage)) { throw new IllegalArgumentException("Cannot create HTTP(S) connection. Accept-Language header is not set."); } if (StringUtils.isEmpty(acceptEncoding)) { throw new IllegalArgumentException("Cannot create HTTP(S) connection. Accept-Encoding header is not set."); } // Set headers conn.setRequestProperty("User-Agent", userAgent); conn.setRequestProperty("Accept", accept); conn.setRequestProperty("Accept-Language", acceptLanguage); conn.setRequestProperty("Accept-Encoding", acceptEncoding); conn.setRequestProperty("Connection", "keep-alive"); conn.setRequestProperty("Upgrade-Insecure-Requests", "1"); if (cookies != null) { conn.setRequestProperty("Cookie", cookies); } this.connection = conn; }
However I still receive:
java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:209) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) at cz.wm.common.net.HttpConnection.retrieveContent(HttpConnection.java:73) at cz.wm.agent.slave.job.search.AbstractSearchJob.getFinalURL(AbstractSearchJob.java:192) at cz.wm.common.job.AbstractAgentJob.run(AbstractAgentJob.java:46) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
This are, for instance, the problematic sites that I can open via browser but can't open through java HttpConnection:
- https://www.selavita.cz/produkty/anglicka-slanina-detail-215?variant=538&gclid=CODG5JqP2dQCFW8o0wodgT0NLg
- https://www.koupelny-radiatory.cz/lotta-umyvadlova-pakova-baterie-bez-vypusti-chrom?gclid=CJvI9Yya2dQCFQEL0wodhH0IuQ
- https://www.pracovniochrana.cz/pracovni-kalhoty-orion-teodor-sedo-cerne-prodlouzene?gclid=cojyvugz2dqcfuko0wodaxkc2w
- https://www.fitness4u.cz/viridian-organic-joint-omega-oil-200ml/