JSON array to bash variables using jq

Question

I've got a JSON array like so:

{ "SITE_DATA": { "URL": "example.com", "AUTHOR": "John Doe", "CREATED": "10/22/2017" } } 

I'm looking to iterate over this array using jq so I can set the key of each item as the variable name and the value as it's value.

Example:

• URL="example.com"
• AUTHOR="John Doe"
• CREATED="10/22/2017"

What I've got so far iterates over the array but creates a string:

constants=$(cat ${1} | jq '.SITE_DATA' | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]") 

Which outputs:

URL=example.com AUTHOR=John Doe CREATED=10/22/2017 

I am looking to use these variables further down in the script:

echo ${URL} 

But this echos an empty output at the moment. I'm guessing I need an eval or something in there but can't seem to put my finger on it.

Answer 1

Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.

All of this is only sensible if you completely trust the input data (e.g. it's generated by a tool you control). There are several possible problems otherwise detailed below, but let's assume the data itself is certain to be in the format you expect for the moment.

You can make a much simpler version of your jq program:

jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' 

which outputs:

URL='example.com' AUTHOR='John Doe' CREATED='10/22/2017' 

There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including appropriate quotes & escaping around the value with @sh.

All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.

This string is evalable if you completely trust the input data and has the effect you want:

eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' < data.json)" echo "$AUTHOR" 

---

As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong. In particular, if the key contains shell metacharacters like $ or whitespace, this could create a running command. It could also overwrite, for example, the PATH environment variable unexpectedly.

If you don't trust the data, either don't do this at all or filter the object to contain just the keys you want first:

jq '.SITE_DATA | { AUTHOR, URL, CREATED } | ...' 

You could also have a problem in the case that the value is an array, so .value | tostring | @sh will be better - but this list of caveats may be a good reason not to do any of this in the first place.

---

It's also possible to build up an associative array instead where both keys and values are quoted:

eval "declare -A data=($(jq -r '.SITE_DATA | to_entries | .[] | @sh "[\(.key)]=\(.value)"' < test.json))" 

After this, ${data[CREATED]} contains the creation date, and so on, regardless of what the content of the keys or values are. This is the safest option, but doesn't result in top-level variables that could be exported. It may still produce a Bash syntax error when a value is an array, or a jq error if it is an object, but won't execute code or overwrite anything.

Answer 2

Building on @Michael Homer's answer, you can avoid a potentially-unsafe eval entirely by reading the data into an associative array.

For example, if your JSON data is in a file called file.json:

#!/bin/bash typeset -A myarray while IFS== read -r key value; do myarray["$key"]="$value" done < <(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + .value ' file.json) # show the array definition typeset -p myarray # make use of the array variables echo "URL = '${myarray[URL]}'" echo "CREATED = '${myarray[CREATED]}'" echo "AUTHOR = '${myarray[URL]}'" 

Output:

$ ./read-into-array.sh declare -A myarray=([CREATED]="10/22/2017" [AUTHOR]="John Doe" [URL]="example.com" ) URL = 'example.com' CREATED = '10/22/2017' AUTHOR = 'example.com' 

Answer 3

Just realized that I can loop over the results and eval each iteration:

constants=$(cat ${1} | jq '.SITE_DATA' | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]") for key in ${constants}; do eval ${key} done 

Allows me to do:

echo ${AUTHOR} # outputs John Doe 

Answer 4

I really like the @Michel suggestion. Sometimes, you may really just extract some variables value to execute a task in that specific server using Bash. So, if desired variables are known, using this approach is the way to avoid multiple calls to jq to set a value per variable or even using the read statement with multiple variables in which some can be valid and empty, leading to a value shift (that was my problem).

My previous approach that lead will lead to a value shift error – if .svID[ ].ID="", sv will get the slotID value.

-rd '\n' getInfo sv slotID <<< $(jq -r '(.infoCMD // "no info"), (.svID[].ID // "none"), (._id // "eeeeee")' <<< $data) 

If you downloaded the object using curl, here is my approach to rename some variables to a friendly name as extract data from data arrays.

Using eval and filters will solve the problem with one line and will produce variables with the desired names.

eval "$(jq -r '.[0] | {varNameExactasJson, renamedVar1: .var1toBeRenamed, varfromArray: .array[0].var, varValueFilter: (.array[0].textVar|ascii_downcase)} | to_entries | .[] | .key + "=\"" + (.value | tostring) + "\""' <<< /path/to/file/with/object )" 

The advantage in this case, is the fact that it will filter, rename, format all the desired variables in the first step. Observe that in there is .[0] | that is very common to have if the source if from a RESTful API server using GET, response data as:

[{"varNameExactasJson":"this value", "var1toBeRenamed: 1500, ....}] 

---

If your data is not from an array, i.e., is an object like:

{"varNameExactasJson":"this value", "var1toBeRenamed: 1500, ....} 

just remove the initial index:

eval "$(jq -r '{varNameExactasJson, renamedVar1: .var1toBeRenamed, varfromArray: .array[0].var, varValueFilter: (.array[0].textVar|ascii_downcase)} | to_entries | .[] | .key + "=\"" + (.value | tostring) + "\""' <<< /path/to/file/with/object )"