How to integrate multiple Python/Perl scripts to be executed in one bash script?

Question

I have the following Perl scripts (though this applies to Python and other scripting languages): script1.pl, script2.pl, script3.pl

The way these scripts where written, users execute them with input flags, and the output is a file saved.

perl script1.pl --i input1.tsv ## this outputs the file `outputs1` perl script2.pl --i outputs1 ## this outputs the file `outputs2` perl script3.pl --i outputs2 ## this outputs the file `final_output` 

Now, I would like to create an executable bash script that allows users to simply work with input1 and get the output in return final_output.

Here's how I would do this with only one perl script, execute.sh:

#!/bin/sh source ~/.bash_profile FLAG1="--i=$1" perl script1.pl $FLAG1 

which could be run on the command line execute.sh input1.tsv

For my example with three scripts, how would I pipe the intermediate outputs into the intermediate scripts to created one execute.sh script, e.g. outputs1 into script2.pl, then outputs2 into scripts3.pl, etc.?

Isn't there a way for me to do this without rewriting the perl/python scripts?

Answer 1

The 3 lines you wrote to demonstrate what you want are a valid bash script: a script is nothing more than a sequence of commands executed one after the other by a shell.

The only additional code you might want would be something to clean up the temporary files produced by script1.pl and script2.pl. Further, you might want to use absolute paths to specify where script{1,2,3}.pl live.

#!/bin/sh trap 'rm outputs1 outputs2' EXIT perl script1.pl --i "$1" perl script2.pl --i outputs1 perl script3.pl --i outputs2 

$1 represents the value of the first argument passed to the script, so it would be called with

execute.sh input1.tsv 

as desired.

You can't really set up pipes between the three scripts, because they appear to be hard-coded to produce specific output files.

Answer 2

Assuming the output files are created locally, ie. in the directory where the script is located, you can try to run the scripts in a temporary working directory, to which you can copy the scripts before executing them, and in which no other processes will create files. Then you will be able to control and operate on the new files created by particular scripts.

You can create the temporary directory in /tmp or your active user directory. Add to the end of your script a command that will tear down the directory. If you want any output files to be saved elsewhere, you can copy them at the right time to the right place.