Simple Bash script Nested while and for loops while reading from a text file

Question

I have the following bash script which works fine but only goes through the for loop once!

#!/bin/bash csv_file="regime.csv" ratio=(0.2 0.4 0.6 0.8) while IFS=',' read -r col1 col2 col3 col4; do echo $col1 echo $col2 echo $col3 echo $col4 for rat in "${ratio[@]}"; do dir_name="${col4}_T${col1}_erate${col2}_${rat}" mkdir -p "$dir_name" cp in.C3N run.sh BNC.tersoff data_C3N.txt "$dir_name/" sed -i "s/\(variable[[:space:]]\+T equal \).*/\1${col1}/" "$dir_name/in.C3N" sed -i "s/\(variable[[:space:]]\+erate equal \).*/\1${col2}/" "$dir_name/in.C3N" sed -i "s/\(variable[[:space:]]\+reg equal \).*/\1${rat}/" "$dir_name/in.C3N" sed -i "s/\(variable[[:space:]]\+time equal \).*/\1${col3}/" "$dir_name/in.C3N" if [ "$col4" = "ZZ" ]; then sed -i '/^fix[[:space:]]\+deform Boundary deform 1 y erate/{ N s/\(fix.* y erate.*\)\n\(fix.* x erate.*\)/\2\n\1/ }' "$dir_name/in.C3N" fi # Change into the new directory, run run.sh, and return to the parent folder. pushd "$dir_name" > /dev/null ./run.sh popd > /dev/null done done < "$csv_file"

I can't find where the logic is wrong. So it goes through the ratio values 0.2, 0.4, 0.6, 0.8 and then ends. The csv file has 30 lines with 4 columns.

I have this sample code which works fine and I don't know what the difference is.

#!/usr/bin/env bash file="test.txt" a=(1 2 3) while IFS=' ' read -r col1 col2 rest; do for val in "${a[@]}"; do echo "$col1 $col2 and $val" done done < "$file"

Thank you.

** Update 1:**

The regime.csv file is created in Ubuntu using Jupyter-notebook.Here is the output of:

cat -A regime.csv | head -5 100,0.01,978,AC$ 1500,0.01,338,AC$ 300,0.01,780,AC$ 1000,0.01,425,AC$ 1500,0.001,2901,AC$

Update 2:

You can't technically run the files because you need LAMMPS installed but and it is a pain to do so, but you can replace the mpirun line in the run.sh file to do a simple echo or something.

run.sh:

#!/bin/bash LAMMPS_EXECUTABLE='/home/km9mb/Downloads/lammps-29Aug2024/src/lmp_mpi' INPUT_FILE="in.C3N" mpirun --oversubscribe -np 32 ${LAMMPS_EXECUTABLE} -in ${INPUT_FILE}

UPDATE 3

It is definitely the run.sh file that is causing the problem. When I comment out the "./run.sh" the whole thing works just fine! Does anyone knows how to fix this?

SOLVED!!!!

Thanks to @pjh I was able to fix it by adding "--stdin none" flag to my "mpirun" command in "run.sh" file! It turned out that run.sh was eating my stdin terminating the while loop!

First guess would be the regime.csv has windows line endings, Or you need to cd in a subshell. — Jetchisel, CommentedApr 18 at 23:35
the second/sample code block is working with a different file which would mask any potential issues with regime.csv; consider making a copy of the main script, comment out everything inside the for loop and add something like echo "for:$col1:$rat" then run the script; if the script loops successfully then incrementally uncomment some lines in the for loop and run the script again; repeat until you find what's breaking the for loop; also, by any chance does run.sh modify regime.csv? — markp-fuso, CommentedApr 18 at 23:54
fwiw, I cut-n-pasted your (first) script into my environment, created a 6-line 4-field regime.csv file (2 of the lines end in ,ZZ), made a run.sh that simply does echo 'hi', and ran touch on the other data files; ran the script and it cycled through both the while and for loop as expected while creating a slew of *erate* subdirectories (with each subdirectory containing the 4 cp'd files) — markp-fuso, CommentedApr 19 at 0:04
if you continue to have problems troubleshooting the script then you'll need to look at providing us with an MRE; in particular you'd need to provide us with (minimal) data files and scripting that demonstrates the problem — markp-fuso, CommentedApr 19 at 0:07
One possibility is that something (e.g. ssh or ffmpeg) in your run.sh reads standard input and thus consumes all the lines that should be read by the loop. See While loop stops reading after the first line in Bash. Also see the How to keep other commands from "eating" the input section of BashFAQ/001 (How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?). — pjh, CommentedApr 19 at 0:21

3aska · Accepted Answer · 2025-04-19 17:57:10Z

I think I found your issue. The problemis that your shell script processing CSV files. The loop is only processing the first line because of how your CSV file is likely formatted.

The most common culprit is Windows-style line endings (CRLF instead of just LF). When bash reads these files it sometimes treats the whole file as a single line or has other strange behavior.

Try this modified version:

#!/bin/bash csv_file="regime.csv" ratio=(0.2 0.4 0.6 0.8) while IFS=, read -r col1 col2 col3 col4; do # Strip potential carriage returns here col4=${col4//$'\r'/} echo "Processing: $col1, $col2, $col3, $col4" for rat in "${ratio[@]}"; do dir_name="${col4}_T${col1}_erate${col2}_${rat}" mkdir -p "$dir_name" cp in.C3N run.sh BNC.tersoff data_C3N.txt "$dir_name/" sed -i "s/\(variable[[:space:]]\+T equal \).*/\1${col1}/" "$dir_name/in.C3N" sed -i "s/\(variable[[:space:]]\+erate equal \).*/\1${col2}/" "$dir_name/in.C3N" sed -i "s/\(variable[[:space:]]\+reg equal \).*/\1${rat}/" "$dir_name/in.C3N" sed -i "s/\(variable[[:space:]]\+time equal \).*/\1${col3}/" "$dir_name/in.C3N" if [ "$col4" = "ZZ" ]; then sed -i '/^fix[[:space:]]\+deform Boundary deform 1 y erate/{ N s/\(fix.* y erate.*\)\n\(fix.* x erate.*\)/\2\n\1/ }' "$dir_name/in.C3N" fi pushd "$dir_name" > /dev/null ./run.sh popd > /dev/null done done < "$csv_file"

As a quick test you could also run this command to check if your file has Windows line endings:

cat -A regime.csv | head -5

If you see ^M$ at the end of lines instead of just $, then it's definitely a line ending issue.

Alternatively, you could convert the file first:

dos2unix regime.csv

Also worth checking if your CSV has any empty lines or commented lines that might be causing issues. I'd add those debug echoes like I did above to see exactly what's being processed.

Thank you, I tried everything you said with no luck! Specifically, the files are all created and edited in Ubuntu. even though I double checked and used dos2unix on them. I have updated the question for you to take a look at. — Karen Tabar, CommentedApr 19 at 20:11
It turns out there is something going on with the run.sh causing the problem. I don't know what the problem is or how to fix it. Please see update 3 in the original question. — Karen Tabar, CommentedApr 19 at 20:26
CRLF line endings would not cause the script to read the whole file at once, it'd just cause each line the script read to end with CR since the script would treat LF as the newline either way. Empty lines or commented lines also would not cause the problem the OP is having. — Ed Morton, CommentedApr 21 at 12:48
@EdMorton Is that so? Can you show us the truth of this information? — 3aska, CommentedApr 21 at 15:04
Sure. Execute IFS= read -r line < <(printf 'foo\r\n'); printf '<%s>\n' "$line" | cat -v and it'll output <foo^M> because it behaves exactly as I stated and that control-M represents the CR in the output. There's nothing special about CRs, they're just one more character to Unix tools. For more info see why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it. As for empty or commented lines, Idk why youd think theyd cause issues like the OP sees so idk what to say to disprove it — Ed Morton, CommentedApr 21 at 15:34

Collectives™ on Stack Overflow

Simple Bash script Nested while and for loops while reading from a text file

1 Answer 1

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Linked

Related