My sequential bash script shows non-sequential behavior

Question

I have a problematic bash script and I really appreciate your help in identifying the problem. The bash script Script.sh is as follows:

#!/bin/bash ./command1 -i inFile.txt -o outFile1.bin > log1.txt ./command2 -i inFile.txt -o outFile2.txt > log2.txt ./command3 -i outFile2.txt -o outFile3.txt > log3.txt ... 

Here, -i defines the input file and -o defines the output file. There are some dependencies, for example the output file of command 2 is used as the input file for command 3.

I call the script using

nohup ./Script.sh > scriptLog.txt 2> scriptErr.txt & 

Here is the strange thing that happens: the script runs to completion without any errors (scriptErr.txt is empty). The output files (outFile1.txt, outFile2.txt and outFile3.txt) are produced and are correct as much as I can say. The jobs command shows that the run is complete (Done). The command

ps aux | grep ./Script.sh 

returns nothing (only the grep itself). However, the command

ps aux | grep ./command1 

shows that command1 is still running (that is not the case about command2 and command3). And here is what confuses me:

This script is supposed to be sequential. How is it possible that commands 2 and 3 are completed while command 1 is not completed yet?

If command 1 is not completed, then how is that the output file is correctly produced?

Thank you very much!

Answer 1

The problem here comes from nohup + &.

You use those to send the process to background. & also forks you current process. So instead of out Script.sh you get two (or multiplied by number of times you use &). New Script.sh process are children of the process you started. commandX is started from forked Script.sh.

Once main script process is finished, the parent Script.sh is gone, all the children are moved to init process. Those are different processes, just the name match and might introduce some confusion.

Answer 2

It's easy to construct a script that stays running despite returning:

$ ./still_running I'm done $ ps aux | grep '[s]till_running' anthony 25666 0.0 0.0 7420 2092 pts/31 S 16:43 0:00 /bin/bash ./still_running 

That script is fairly trivial:

#!/bin/bash ( sleep 5m echo 'done now' ) & echo "I'm done" 

It starts a subshell, which it runs in the background (due to the the ampersand), and it stays running even when the parent process exits.

It's impossible to say for sure what's happening with your script without seeing the script (ideally a minimal test case), but this or something similar is a possibility.