Running python script many times in a bash file

Question

I want to run a python script many times with various arguments. To do that, I've written the following bash script:

requests=(25 50 75 100) factors=(3 6) graphsizes=(25 50 75) for request in "${requests[@]}"; do for factor in "${factors[@]}"; do for size in "${graphsizes[@]}"; do echo "Now Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out; echo nohup python3 -u main.py "$request" 50 "$factor" "$size" > ${request}_${factor}_${size}.log & echo "Done Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out; done done done 

I added ; at the end of the first and last echo because I do not want them to be run in parallel. In fact I want every call to the python script main.py to be run sequentially not in parallel, since the script itself is already parallelized and don't want any race conditions.

I know normally we use a ; to make jobs run sequentially, but if I do that after the & in the nohup line, I get the error

syntax error near unexpected token `;' 

How do I make each iteration of the loop run sequentially?

Answer 1

The way you have written the script, on multiple lines, the ';' is implied by the linebreak. You can remove the ';' at the end of each of the three lines inside the innermost for loop and the script will run sequentially. You only need the ';' if you move things onto the same line when there should be a break between them, such as when you're putting the do on the same line as your for specification. Note the alternate placement of the do in the rewritten script below:

for request in "${requests[@]}" do for factor in "${factors[@]}" do for size in "${graphsizes[@]}" do echo "Now Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out echo nohup python3 -u main.py "$request" 50 "$factor" "$size" > ${request}_${factor}_${size}.log echo "Done Running: n = ${request}, factor = ${factor}, size = ${size}" >> nohup.out done done done 

Answer 2

When then number of combinations gets bigger, you will overwhelm your computer.

This is where GNU Parallel comes in handy:

nohup parallel -j100% --header : python3 -u main.py {request} 50 {factor} {size} '>' {request}_{factor}_{size}.log \ ::: request 25 50 75 100 \ ::: factor 3 6 \ ::: size 25 50 75 

Change -j100% if you do not want run one job per CPU core.