Bash script to check whether python scripts are running in screens - restart if not

Question

Hello so I'm currently trying to set up a bash script that starts 5 python processes in 2 separate screens, then checks whether my python scripts are currently running, and restarts them if they aren't. There are 5 python scripts all together. 4 scripts are ran in 1 screen and 1 script in another. I am new to linux and scripting so I'm hoping I've just made a small mistake. I'm currently just trying to kill them all and then restart them all in screens if they any of them have stopped. Here's my bash script:

#!/bin/bash screen -dm -S "screen1" python script1.py screen -dm -S "screen2" python script2.py & python script3.py & python script4.py & python script5.py while true; do num_procs=$(pgrep -lf python | wc -l) if [ "$num_procs" != "7" ]; then pkill python screen -dm -S "fail" python script_failed.py sleep 10 pkill python screen -dm -S "screen1" python script1.py screen -dm -S "screen2" python script2.py & python script3.py & python script4.py & python script5.py fi sleep 20 done 

I also want to be notified if my code fails so I can retrace why it did, which is why I start the script_failed.py because it sends me an email that it failed. I used num_procs = 7 because there's 5 python processes and 2 screens. For some reason the screens also count towards the num_procs. As you can see below in my output for pgrep -af python:

1146 SCREEN -dm -S screen1 python scrip1.py 1148 python script1.py 1154 python script3.py 1155 python script4.py 1156 python script5.py 1157 SCREEN -dm -S screen2 python script2.py 1158 python script2.py 

I tried to use echos around the place to see where the code was getting to and getting stuck but I found no echos even work after I've done the initially started the screens. Also the while loop seems to not work if I've killed just 1 of the python scripts, but if I kill them all it does work and does restart. Could the echos and while loop be getting lost in a detached screen or something until I pkill all the process? I've also noticed that pkill sometimes doesn't kill the python processes.

Any help would be appreciated. Thanks.

Answer 1

I would probably do something like what I've outlined below. This will create a separate watchdog script for each screen instance so you don't have to completely restart the screen instances, and by using .pid files would allow more accurate verification of whether the scripts are still running.

Edit each python script (1-5) to create a .pid file (example: https://stackoverflow.com/a/789383/5670331)

Edit the main script:

#!/bin/bash screen -dm -S "screen1" watchdog1.sh screen -dm -S "screen2" watchdog2.sh 

watchdog1.sh

while true; do sleep 10 ps up `cat /tmp/python_script_1.pid` >/dev/null && echo "Script 1 is running" || python script1.py & done 

watchdog2.sh

while true; do sleep 10 ps up `cat /tmp/python_script_2.pid` >/dev/null && echo "Script 2 is running" || python script2.py & ps up `cat /tmp/python_script_3.pid` >/dev/null && echo "Script 3 is running" || python script3.py & ps up `cat /tmp/python_script_4.pid` >/dev/null && echo "Script 4 is running" || python script4.py & ps up `cat /tmp/python_script_5.pid` >/dev/null && echo "Script 5 is running" || python script5.py & done