Curl Command (Beginners Linux)

Question

I am attempting to use the CURL command to download the latest zip file (as of today it is FuelWatchRetail-03-2020.csv.zip) from this site.

The command I currently have is as follows:

curl https://www.fuelwatch.wa.gov.au/fuelwatch/pages/public/historicalFileDownloadRetail.jspx#/FuelWatchRetail-03-2020.csv.zip -o new.zip 

The file that is output appears to be the html from the initial webpage. I can not seem to find a link to the file location I am trying to download. I have googled extensively to find a solution, however I am obviously not searching for the correct thing.

Answer 1

If you try to retrieve this file using a regular browser with Developer console launched, you'll notice that, when you click on the "link", it will trigger a HTTP POST request to https://www.fuelwatch.wa.gov.au/fuelwatch/pages/public/historicalFileDownloadRetail.jspx with the following parameters:

{ "j_idt72":"j_idt72", "j_idt72:resultsTbl_rppDD":"20", "javax.faces.ViewState":"-4860850130551349268:6438609436745021067", "j_idt72:resultsTbl:0:j_idt75":"j_idt72:resultsTbl:0:j_idt75" } 

In return, the requested file is given as attachment:

{"headers": [ { "name":"Content-Disposition", "value":"attachment;filename=\"FuelWatchRetail-03-2020.csv.zip\"; filename*=UTF-8''FuelWatchRetail-03-2020.csv.zip" }, { "name":"Content-Type","value":"application/pdf"}, [remaining data are useless for this explanation] } 

So it's quite impossible to retrieve the file you want without implementing a bit of "logic" around curl. It'll probably be less painfull to implement this using a scripting language with HTTP/HTML abilities such as Perl, Python, etc...