Bash variable substitution with json string as variable within larger json object

Question

I am having trouble getting the AWS cli working nicely with a bash variable. The AWS cli command takes in a JSON string as a parameter. However, within this json string I am using a variable $SCHEMA_DEFINITION which itself is a JSON string. For some reason the double quotes are being removed from $SCHEMA_DEFINITION and making it invalid JSON. How can I preserve these quotes and still produce a valid AWS command?

aws glue create-schema --cli-input-json '{"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName": "'$modelClassName'","DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": "'"$SCHEMA_DEFINITION"'"}' 

Gives the error:

An error occurred (InvalidInputException) when calling the CreateSchema operation: Schema definition of AVRO data format is invalid: Unexpected character ('n' (code 110)): was expecting double-quote to start field name at [Source: (String)"{name:hello}" 

Here is how I set the variable: SCHEMA_DEFINITION={"name":"Hello"}

Answer 1

The JSON you seem to want to create should contain a JSON object encoded as a JSON string as the value for the SchemaDefinition key. You also seem to want to include the value of the shell variable modelClassName as the value for the SchemaName key.

You can create your JSON using jq like so:

json=$( jq -n -c \ --arg SchemaName "$modelClassName" \ --arg SchemaDefinition '{"name": "Hello"}' \ '{"RegistryId":{"RegistryName":"hxp-schema-registry"},"DataFormat":"AVRO","Compatibility":"FULL_ALL"} + $ARGS.named' ) 

or, for easier reading,

json=$( jq -n -c \ --arg SchemaName "$modelClassName" \ --arg SchemaDefinition '{"name": "Hello"}' ' { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "DataFormat": "AVRO", "Compatibility": "FULL_ALL" } + $ARGS.named' ) 

or even,

schema_json=$( jq -n -c --arg name 'hello' '$ARGS.named' ) json=$( jq -n -c \ --arg SchemaName "$modelClassName" \ --arg SchemaDefinition "$schema_json" ' { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "DataFormat": "AVRO", "Compatibility": "FULL_ALL" } + $ARGS.named' ) 

This correctly encodes both the schema definition JSON object and the model class name as JSON strings and inserts them in the appropriate place in the JSON document.

Then call your command:

aws glue create-schema --cli-input-json "$json" 

Answer 2

The shell is eating your quotes. Quotes are special for the shell, it needs them to protect other special characters like spaces:

$ var=aaa bb cc bash: bb: command not found terdon@tpad ~ $ var="aaa bb cc" terdon@tpad ~ $ echo "$var" aaa bb cc 

So, in your case, the shell thinks the quites are there to protect the value so it doesn't save them as part of the variable's value:

$ SCHEMA_DEFINITION={"name":"Hello"} $ echo "$SCHEMA_DEFINITION" {name:Hello} 

But don't worry, there's an easy fix! You can use single quotes around the variable definition:

$ SCHEMA_DEFINITION='{"name":"Hello"}' $ echo "$SCHEMA_DEFINITION" {"name":"Hello"} 

Or you can escape the quotes:

$ SCHEMA_DEFINITION={\"name\":\"Hello\"} $ echo "$SCHEMA_DEFINITION" {"name":"Hello"} 

Or you can use the printf builtin command:

$ printf -v SCHEMA_DEFINITION '{"%s":"%s"}' "name" "Hello" $ echo "$SCHEMA_DEFINITION" {"name":"Hello"} 

Or you can use the jo(1) utility:

SCHEMA_DEFINITION=$(jo name=hello) 

I would even suggest building the entire thing as a variable first:

schema_definition='{"name":"Hello"}' modelClassName='"thisModel"' json_string='{"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName":' json_string="$json_string $modelClassName" json_string="$json_string"' "DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": '"$schema_definition" 

Which results in:

$ echo "$json_string" {"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName": "thisModel" "DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": {"name":"Hello"} 

So you can now just do:

aws glue create-schema --cli-input-json "$json_string" 

Answer 3

Terdon is right in that your json variable was set incorrectly but you are also passing it to the command incorrectly.

You are passing:

{ "RegistryId": { "RegistryName": "hxp-schema-registry" }, "SchemaName": "'$modelClassName'", "DataFormat": "AVRO", "Compatibility": "FULL_ALL", "SchemaDefinition": "'"$SCHEMA_DEFINITION"'" } 

Which would expand to:

{ "RegistryId": { "RegistryName": "hxp-schema-registry" }, "SchemaName": "'$modelClassName'", "DataFormat": "AVRO", "Compatibility": "FULL_ALL", "SchemaDefinition": "{"name":"Hello"}" } 

Note specifically "{"name":"Hello"}".

I would recommend using a heredoc for this but either way you need to remove the extra double quotes around the {}.

read -rd json <<EOF { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "SchemaName": "$modelClassName", "DataFormat": "AVRO", "Compatibility": "FULL_ALL", "SchemaDefinition": $SCHEMA_DEFINITION } EOF 

Or using jo(1):

json=$(jo RegistryId=$(jo RegistryName=hxp-schema-registry) \ SchemaName=$modelClassName \ DataFormat=AVRO \ Compatibility=FULL_ALL \ SchemaDefinition=$SCHEMA_DEFINITION) 

Then you would call your command with a variable:

aws glue create-schema --cli-input-json "$json" 

---

Also I recommend https://jsonlint.com/ for validating your json inputs if you are running into issues like this.