cURL inside script

Question

I'm trying to troubleshoot some problem on FreeBSD server that is intermittently (at random times) being blocked by a network firewall from downloading some file over HTTPS from a webserver.  In an effort to understand what happens when the block happens, I need to run the following cURL:

curl --interface nic1 --verbose --insecure 'https://speed.hetzner.de/100MB.bin' > /dev/null 

and then read the stderr of the command.  However, since the problem happens on random times that cannot be expected, I want to run this curl once every 10 seconds and store the stderr in a file so that I can review it later, I used the following script (named cURLScript) to achieve this:

#!/bin/sh while true do date echo "=================================" curl --interface nic1 --verbose --insecure 'https://speed.hetzner.de/100MB.bin' > /dev/null 2>&1 echo "" sleep 10 done 

I then run the script:

/bin/sh cURLScript > log & 

I expect that log file would contain the stderr, however, it does not; the log is missing the stderr:

Svr12loc:service 21] cat log Thu Mar 24 16:42:34 CET 2022 ================================= Thu Mar 24 16:43:03 CET 2022 ================================= Thu Mar 24 16:43:33 CET 2022 ================================= Thu Mar 24 16:44:02 CET 2022 ================================= Thu Mar 24 16:44:32 CET 2022 ================================= Thu Mar 24 16:45:00 CET 2022 ================================= 

Can someone suggest what is it I'm doing wrong here?

Answer 1

Your question is unclear.  I guess you are saying that

• you don’t care about the HTML output from cURL (which you’re sending to HTMLFILE),
• you care about the “verbose” messages;
• the verbose messages are going to the terminal (and not the Log file), and
• you want them to go to the Log file (and not the terminal).

In Unix (and Unix-like operating systems, and even some others), programs have two automatic output streams: standard output (abbreviated “stdout”) and standard error (abbreviated “stderr”).  As the names suggest, stdout is meant for the primary output, and stderr is meant for error messages and diagnostics.  By default, both of these point to the terminal.  When you say > HTMLFILE, you are redirecting cURL’s stdout to HTMLFILE.  But the verbose messages are being written to stderr.

What you want is to send cURL’s stderr to the stdout of the script (which is the Log file).

This is easy in any of the standard, POSIX-compliant shells derived from the Bourne shell:

#!/bin/sh while true do date echo "=================================" curl --interface nic1 --verbose --insecure https://www.20min.ch  2>&1  > HTMLPAGE echo "" sleep 10 done

2>&1 does exactly what I said above — redirect the stderr to the stdout.

It’s trickier in C shell:

#!/bin/csh set t=`mktemp` while ((1)) date echo "=================================" ( curl --interface nic1 --verbose --insecure https://www.20min.ch > HTMLPAGE ) >& "$t"cat "$t" echo "" sleep 10 end

In csh, >& means send stdout and stderr to the following file.  Sending them separate places (which is what you want) is tricky.  The trick is to send the stdout where you want it, then wrap the command in parentheses, and then use >& on the wrapped command.  I don’t know how to simply send the stderr to the stdout.  (There probably is a way, and I anticipate being told what it is.)  Above, I used the trick of sending the stderr to a temporary file, and then sending the file to stdout with cat.

I encourage you to write scripts in the standard shell.  The C shell is not good for scripting.

P.S. If you don’t want to save the standard output from curl, why not send it to /dev/null?

Answer 2

You're sending the output of curl to the file HTMLPAGE. The > is a redirection operator defined to do exactly that.

If you want the page contents to be included in your script output (i.e. in the log file) don't redirect it on the line calling curl