Logging the output of remote commands on multiple ssh servers without delay

Question

I've written a simple bash script that ssh's into 3 hosts (2 remote, 1 my own for testing) and runs a long running gui program that outputs text to the terminal that I'd like to log.

#!/bin/bash ssh -f -X user@remote1 '(python -u long_running_program.py &> ~/log1.txt)' ssh -f -X user@remote2 '(python -u long_running_program.py &> ~/log2.txt)' ssh -f -X user@localhost '(python -u long_running_program.py &> ~/log3.txt)' multitail -s 3 ~/log* 

From the above, ssh is called with the f parameter to run in the background and allow the script to continue execution. On the remote server, the python program is called with the unbuffered switch and the output redirected to a log file (note that the home directories of the remote and local machines are all on a network mounted drive so using ~ gives the same path on all of them).

The above technically works, however the 2 logs (log1,log2) from the remote machine are very slow to update. It can be 20+seconds before the file is updated (regardless of if you are using multitail, vi, etc to view the file), while the local log file (log3) update is immediate.

To confirm this should not be happening, I can manually ssh into the remotes and run the program without redirecting the output to file. The output is streamed continuously without delay, as expected. I have also tried various things like trying to turn off buffering and using script/tee/etc to no avail.

Does anyone know what's causing this and/or how to resolve it?

Answer 1

I suggest using pssh which was specifically designed for executing multiple parallel SSH sessions, instead of running several serial SSH commands concurrently.

E.g.,

pssh -i -H user@remote1 -H user@remote2 -H user@localhost '(python -u long_running_program.py &> ~/log$PSSH_NODENUM.txt)'