Curl Parallel requests using links source file

Question

I have this script to go through a list of URLs and the check return codes using Curl.

Links file goes like this:

https://link1/... https://link2/... https://link200/... (...) 

The script:

INDEX=0 DIR="$(grep [WorkingDir file] | cut -d \" -f 2)" WORKDIR="${DIR}/base" ARQLINK="navbase.txt" for URL in $(cat $WORKDIR/$ARQLINK); do INDEX=$((INDEX + 1)) HTTP_CODE=$(curl -m 5 -k -o /dev/null --silent --head --write-out '%{http_code}\n' $URL) if [ $HTTP_CODE -eq 200 ]; then printf "\n%.3d => OK! - $URL" $INDEX; else printf "\n\n%.3d => FAIL! - $URL\n" $INDEX; fi done 

It takes a little while to run through every URL, so I was wondering how to speed those up curl requests. Maybe I could use some parallel Curl requests, but using "xargs" inside a "for" loop while also printing a message doesn't seem the way to go.

I was able to use "xargs" out of the script and it sort of works, although not showing the correct HTTP code.

cat navbase.txt | xargs -I % -P 10 curl -m 5 -k -o /dev/null --silent --head --write-out '%{http_code}\n' % 

I couldn't find a way to insert that into the script.

Any tips?

Answer 1

Curl 7.68 has the --parallel, --parallel-max and --parallel-immediate options. This would be the cleanest approach. See release notes here and some man options here .

Answer 2

I'm using cURL v8.7.1 and that supports the --parallel option. If you're on a SLURM cluster, make sure to load the correct cURL binary before launching downloads.

E.g.,

module load cURL/8.7.1