Bash: split multi line input into array

Question

I've got a file with strings and base64 encoded data over multiple lines, that are sepearated by a comma.

Example:

1,meV9ivU4PqEKNpo5Q2u2U0h9owUn4Y8CF83TTjUNWTRQs7dEgVxnsMgf4lvg9kvxcIaM3yB4Ssim z46M/C7YlovNUmrjOByhV1SCb/bGyv1yL7SYFnw1GHbYjdH0b6UZ7nQzJHU6VmwMo0V77vFNy6nx rmJZ4KqW9EcjdV1plQmsVXSiZVi61+fNOHCMDmVtJ4q097geWxf4bT0/k/yRyRwi5Zr8BC64htVS AdwOSo4PIk7xDLOzLywAYOCDQvD/zuErf1L0e8nHGz2LKdApHdEWB7Y2yM3iZyXuQ4sMx0+oX66+ FxwUulvHj+EpXtLJx5rmV7AUjr/GsNw/1aYAGPCfz0S+//Ic5pXX5rY1fZ96oFGw4a9vRiAmxe/w ZOza6LtwuF+WUHjbIeWTUKKQGgFIM81dwVHHY7xdRnQhK5J0Zf3Xz0GzzZj5/2YFbI8q7lVkJ3ZQ 7Oqt0qdfk3aj+BQhOxmn1F55yACPBZoPUw6K8ExTHHGVGdCEiIDTu5qKHcUwK0hGAZA9Mun5KTO0 gPs9JxF8FJjkQBF7rEa6TP3pH5OwdkATH2uf+Zcmp1t6NbBymXVlsLzWZookVsaT1DNXf1I1H8Xz 8dnfh6Yl63jSr2PAhDrcOqJNM8Z9/XhBGxtlD1ela3nq6N1ErR1Gv1YZKNeNcL7O2Z3Vl2oyyDw=,U2FsdGVkX1/c8rTTO41zVT7gB+KL+n7KoNCgM3vfchOyuvBngdXDGjXTvXTK0jz6 

Now, I'd like to split the content into an array, so that each multi-line string is an array element.

I tried to use IFS, but that only reads the first line:

filecontent=$(cat myfile) IFS=',' read -a myarray <<< "$filecontent" 

Result:

$myarray[0] = 1 $myarray[1] = meV9ivU4PqEKNpo5Q2u2U0h9owUn4Y8CF83TTjUNWTRQs7dEgVxnsMgf4lvg9kvxcIaM3yB4Ssim 

Expected:

$myarray[0] = 1 $myarray[1] = meV9ivU4PqEKNpo5Q2u2U0h9owUn4Y8CF83TTjUNWTRQs7dEgVxnsMgf4lvg9kvxcIaM3yB4Ssim z46M/C7YlovNUmrjOByhV1SCb/bGyv1yL7SYFnw1GHbYjdH0b6UZ7nQzJHU6VmwMo0V77vFNy6nx rmJZ4KqW9EcjdV1plQmsVXSiZVi61+fNOHCMDmVtJ4q097geWxf4bT0/k/yRyRwi5Zr8BC64htVS AdwOSo4PIk7xDLOzLywAYOCDQvD/zuErf1L0e8nHGz2LKdApHdEWB7Y2yM3iZyXuQ4sMx0+oX66+ FxwUulvHj+EpXtLJx5rmV7AUjr/GsNw/1aYAGPCfz0S+//Ic5pXX5rY1fZ96oFGw4a9vRiAmxe/w ZOza6LtwuF+WUHjbIeWTUKKQGgFIM81dwVHHY7xdRnQhK5J0Zf3Xz0GzzZj5/2YFbI8q7lVkJ3ZQ 7Oqt0qdfk3aj+BQhOxmn1F55yACPBZoPUw6K8ExTHHGVGdCEiIDTu5qKHcUwK0hGAZA9Mun5KTO0 gPs9JxF8FJjkQBF7rEa6TP3pH5OwdkATH2uf+Zcmp1t6NbBymXVlsLzWZookVsaT1DNXf1I1H8Xz 8dnfh6Yl63jSr2PAhDrcOqJNM8Z9/XhBGxtlD1ela3nq6N1ErR1Gv1YZKNeNcL7O2Z3Vl2oyyDw= $myarray[2] = U2FsdGVkX1/c8rTTO41zVT7gB+KL+n7KoNCgM3vfchOyuvBngdXDGjXTvXTK0jz6 

Could someone help me out here?

Answer 1

Your attempt is close to the actual solution. The relevant flag can be found in the read help:

$ help read ... -d delim continue until the first character of DELIM is read, rather than newline 

It doesn't explicitly mention it, but you can set the delimiter to empty, and thus it'll read until EOF.

$ IFS=, read -d '' -a myarray < myfile $ echo ${myarray[2]} U2FsdGVkX1/c8rTTO41zVT7gB+KL+n7KoNCgM3vfchOyuvBngdXDGjXTvXTK0jz6 

Answer 2

Did you try newline on IFS like IFS=$'\n' read -a myarray <<< "$filecontent"

The reason I am suggesting $'\n' is because bash doesn't interpret escape sequences in string literals.

Answer 3

I'm assuming that in your expected result, the first instance of $myarray[2] should have been $myarray[1].

You can achieve the desired result by stripping the embedded newlines before assinging the file's content to the shell variable:

filecontent=$(tr -d \\n <myfile) 

Answer 4

Use -d '' to read whole file instead of just one line.

 -d delim continue until the first character of DELIM is read, rather than newline 

$ cat file.txt abcd efgh $ IFS=$'\n' read -a ARR < file.txt $ echo ${#ARR[@]} 1 # only 1 element 

$ IFS=$'\n' read -a ARR -d '' < file.txt $ echo ${#ARR[@]} 2 # whole file 

One more thing:
Make sure you don't have set -e in your script if you are using -d '', as the read --help says (at the end):

 Exit Status: The return code is zero, **unless end-of-file is encountered**... 

$ IFS=$'\n' read -a ARR < file.txt $ echo $? 0 $ IFS=$'\n' read -a ARR -d '' < file.txt $ echo $? 1 # Your script will exit if you `set -e` 

Answer 5

Try readarray or its alias mapfile in combination with input redirection.

IFS=',' readarray -t my_array < myfile # Print the array elements echo "${my_array[@]}" 

Huh?

• IFS=',' Sets the field separator to split by comma. You already got this...
• readarray -t my_array This command reads the input and stores each line as an element in the array my_array. The -t option removes the trailing separator from each line.
• < myfile Sends myfile as standard input to readarray