exec command in bash loop for webscrapping

Question

Here is a simple script which is curling https://unix.stackexchange.com/ and storing the result into an array, which is working fine.

#!/usr/local/bin/bash [ -f pgtoscrap ] && { rm pgtoscrap; }; curl -o pgtoscrap https://unix.stackexchange.com/; declare -a arr; fileName="pgtoscrap"; exec 10<&0 exec < $fileName let count=0 while read LINE; do arr[$count]=$LINE ((count++)) done exec 0<10 10<&- 

But, each time I run this script; I get some error for the wrong file descriptor.

./shcrap ./shcrap: line 14: 10: No such file or directory 

I think I don't understand well how to use exec command in a loop correctly. Can someone explain?

-- Update after implementing mapfile for Bash 4 it became much simpler --

#!/usr/local/bin/bash ## Pass a parameter as e.g. ./linkscrapping.bash https://unix.stackexchange.com/ mapfile -t arr < <(curl -s $1); ## Doing exec stuff with process substitution regex="<a[[:print:]]*<\/a>"; ELEMENTS=${#arr[@]}; firstline=0; for((i=0;i<$ELEMENTS;i++)); do if [[ ${arr[${i}]} =~ $regex ]]; then [[ $firstline<1 ]] && { echo ${BASH_REMATCH[0]} > scrapped; let firstline=$firstline+1; } || { echo ${BASH_REMATCH[0]} >> scrapped; } fi done pg2scrap="scrapped"; mapfile -t arr2 < <(cat $pg2scrap); regex="href=[\"\'][0-9a-zA-Z\:\/\.]+"; ELEMENTS2=${#arr2[@]}; line2=0 for ((i=0;i<$ELEMENTS2;i++)); do if [[ ${arr2[${i}]} =~ $regex ]]; then [[ $line2<1 ]] && { echo ${BASH_REMATCH[0]#href=\"} > links; (( line2++ )); } || { echo ${BASH_REMATCH[0]#href=\"} >> links; } fi done; cat links; 

Answer 1

It surely has to do with how you close the file descriptor that you had opened earlier for stdin. Using the below should just be fine

exec 10<&- 

When you do 0<10, you instruct the shell to look for and to slurp in the contents of a file named 10 in your current directory which makes no sense in this context.

In bash you can also use an alternate form exec 10>&- which achieves the same purpose of closing the descriptor.

But that said, you don't need to use exec on random file descriptor and read your input, you can just read in your input with the process substitution technique in bash of form < <() as

while IFS= read -r line; do arr["$count"]="$line" ((count++)) done< <(pgtoscrap) 

Answer 2

exec 10<&0 clones file descriptor number 0 to number 10, effectively saving the original so you can replace the file on fd 0 on the next line. To undo that, you'd need to reverse the numbers, to clone number 10 to number 0, exec 0<&10 (and then close fd 10 with exec 10<&-).

On the other hand, exec 0<10 without the ampersand is just a redirection with a filename 10. Since you don't have such a file, you get an error.

---

That said, you don't need to use exec to temporarily set up a redirection for the while loop. Compound commands can take redirections too, like so:

while read LINE; do ... done < "$filename" 

If you want to read full lines as they are, without whitespace or backslashes affecting the data, you need to unset IFS for read, and use read -r. Also, if you're appending to an array, you don't need to manually keep up with the indexes, you can just use += to append to the array directly:

arr=() # declares it an array and clears it, not strictly necessary though while IFS= read -r line; do arr+=("$line") done < "$filename" 

Or use mapfile (readarray) instead of a manual loop like @BlackJack mentions in comments:

mapfile -t arr < "$filename" 

Or even without a temporary file at all:

#/bin/bash mapfile -t arr < <(curl -s https://unix.stackexchange.com/) 

(Without -t, mapfile leaves the line terminators in place.)