How to have Bash interpret output from command as quoted strings?

Question

I have a program that gets the files selected in the graphical IU (in my case Finder in macOS). The output is something like

'/tmp/file number one.txt' '/tmp/file number two.txt' 

Note the space char in the names, thus the file names are enclosed in ' (single straight ticks)

When using the output of that command in a command substitution in bash for e.g. the ls -l command everything is screwed up. For a test I put the above line into a simple one-liner text file and use it as command line substitution:

$ cat /tmp/files.txt '/tmp/file number one.txt' '/tmp/file number two.txt' $ ls -l $(</tmp/files.txt) ls: "'/tmp/file: No such file or directory ls: '/tmp/file: No such file or directory ls: number: No such file or directory ls: number: No such file or directory ls: one.txt': No such file or directory ls: two.txt'": No such file or directory 

The same happens when I assign the file name string to a variable and use it

$ xxx="'/tmp/file number one.txt' '/tmp/file number two.txt'" $ ls -l $xxx ls: '/tmp/file: No such file or directory ls: '/tmp/file: No such file or directory ls: number: No such file or directory ls: number: No such file or directory ls: one.txt': No such file or directory ls: two.txt': No such file or directory 

Any idea how to solve this? Copying the escaped file names right onto the command line works as expected

$ ls -l '/tmp/file number one.txt' '/tmp/file number two.txt' -rw-r--r-- 1 tester wheel 0B Jul 17 17:21:11 2021 /tmp/file number one.txt -rw-r--r-- 1 tester wheel 0B Jul 17 17:21:16 2021 /tmp/file number two.txt 

My ultimate goal is to use the current Finder selection (which I get through a compiled Applescript) to be available for use in bash. ls is just an example, I might want to use the list of files for tar, cp, mv or any other file handling stuff.

Answer 1

If switching to zsh is an option¹, you could use its z and Q parameter expansion flags that are designed for that:

file_content=$(</tmp/files.txt) quoted_strings=(${(z)file_content}) strings_with_one_layer_of_quotes_removed=("${(Q@)quoted_strings}") ls -ld -- "$strings_with_one_layer_of_quotes_removed[@]" 

Or all in one go:

ls -ld -- "${(Q@)${(z)$(</tmp/files.txt)}}" 

That assumes the syntax of the quoting in the file is compatible with that of zsh.

See also the Z parameter expansion to tweak how the parsing is done. For instance, if the file contains comments (with #) which should be ignored and has more than one line, you'd want:

ls -ld -- "${(Q@)${(Z[Cn])$(</tmp/files.txt)}}" 

See info zsh flags for details.

---

^{¹ I hear zsh is now the default interactive shell in newer versions of macos}

Answer 2

Assume you have this string, literally, with the single quotes embedded:

'/tmp/file number one.txt' '/tmp/file number two.txt' 

You noticed it works correctly when given inline as part of a command line, but not when it comes from an expansion. It doesn't really matter if it's variable expansion or a command substitution, the rules are the same for both. Unquoted expansions undergo word splitting, which you don't want here, since splitting on spaces would split between /tmp/file and number. Quoted expansions don't undergo splitting, but you don't want that either, since presumably you want splitting between the two middle single quotes. Also, there's the fact that quotes resulting from expansions don't quote anything. So, we need to do something different.

Assuming the output is known to be shell syntax, and is safe, you can use eval to have the shell do another round of processing to interpret the quotes:

#!/bin/bash input="'/tmp/file number one.txt' '/tmp/file number two.txt'" eval "ls -ld -- $input" 

or put them in an array for future use:

#!/bin/bash input="'/tmp/file number one.txt' '/tmp/file number two.txt'" eval "files=($input)" for f in "${files[@]}"; do printf "<%s>\n" "$f" done 

Note that if the string going to eval contains unquoted or double-quoted command substitutions (e.g. /dir/$(uname -a), but not '/dir/$(uname -a)'), then your shell will execute the commands involved while processing the eval. Similarly if the string contains an unquoted ) to end the array assignment. So, better make sure to only use this with sources that are under your control.

Also, you do need the double quotes around the string being eval'd, because you don't want it split and globbed before eval gets to process the quotes.

---

There might be ways using other tools to interpret the quotes but not process expansions, e.g. xargs takes quoted strings by default. E.g. this would run printf¹ with each filename as an individual argument:

printf '%s\n' "$input" | xargs printf ":%s:\n" 

Or run ls on them:

printf '%s\n' "$input" | xargs ls -ld -- 

Or you could have xargs run something that then prints the filenames in a simpler format which you can then load to an array in the shell. (This is a bit backward, but I don't know of a way to have Bash just do quote processing without processing expansions.)

#!/bin/bash readarray -td '' files < <( printf '%s\n' "$input" | xargs printf "%s\0") for f in "${files[@]}"; do printf "<%s>\n" "$f" done 

(Here, the printf outputs the strings terminated with NUL bytes, and readarray -td ''² expects output in that format. The NUL being the only value that can't appear in a filename, that's an unambiguous and relatively simple format.)

But note that xargs has a different idea of the exact quoting rules than the shell. It doesn't know about the $'...' style of quoting³, which Bash uses in some cases to output values that contain e.g. embedded newlines, it doesn't recognise backslash inside double quotes⁴... But if the output from Finder is just single quotes (and backslashes to quote any hard single quotes), you might be fine.

---

^{¹ the standalone printf utility, not the printf builtin of your shell, at least once even on empty input (except on some BSDs) and possibly more than once if the list is large}

^{² requires bash 4.4 or above}

^{³ introduced by ksh93 in the 90s}

^{4xargs came with PWB Unix in the late 70s and the quoting syntax is similar to that of the pre-Bourne sh there (Mashey shell), not that of the Bourne shell, let alone ksh93 or bash}

Answer 3

Your best option is to fix whatever's generating such useless file lists so that it generates NUL-separated output instead (because a NUL is the only character that can not be in a path/filename, it is the only separator that is guaranteed to handle any filename with any valid characters). If that's impossible, you can kludge up a "fix" by attempting to convert it to NUL-separated format.

The following perl one-liner will (mostly) convert the file to NUL separated filenames, without quotes surrounding them:

perl -0 -pe "s/'\s+'/\0/sg; s/^'|'\$//sg; s/\x0d?\x0a\$//" file.txt 

The first regex replaces the sequence single-quote, one-or-more whitespace chars, single-quote with NUL characters (the commas and spaces in that are not part of the pattern, they're just grammatical English list separators). The second regex removes the quotes at the beginning and end of the input, and the third removes LF or CRLF at the end of a "line".

This is very far from perfect - some input is un-fixable because there's no way to know with 100% certainty whether a single-quote or a LF is supposed to be embedded in the filename or not (this is why starting with NUL-separated files is the correct solution, not trying to kludge it after the fact).

For example, it will fail if any filenames have an embedded single-quote at either the beginning or end of the filename, or if they have an embedded single-quote followed by one-or-more whitespace characters and followed by another single-quote (e.g. ' ') - all of these will also be replaced with a NUL because of the /g global modifier to the first regex (which is necessary for it to match all filenames in the input instead of just the first). And probably a few other corner-case I haven't thought of yet.

You can redirect the output to another file, feed it into xargs -0r, or use it with the bash built-in readarray and process substitution to populate an array:

readarray -d '' files < <(perl -0 -pe "s/'\s+'/\0/sg; s/^'|'\$//sg; s/\x0d?\x0a\$//" file.txt) 

If you pipe the output into xxd (or hd or hexdump or similar hex-dumper program), you can see that it has become NUL-separated:

00000000: 2f74 6d70 2f66 696c 6520 6e75 6d62 6572 /tmp/file number 00000010: 206f 6e65 2e74 7874 002f 746d 702f 6669 one.txt./tmp/fi 00000020: 6c65 206e 756d 6265 7220 7477 6f2e 7478 le number two.tx 00000030: 74 t