2

I am having trouble getting the AWS cli working nicely with a bash variable. The AWS cli command takes in a JSON string as a parameter. However, within this json string I am using a variable $SCHEMA_DEFINITION which itself is a JSON string. For some reason the double quotes are being removed from $SCHEMA_DEFINITION and making it invalid JSON. How can I preserve these quotes and still produce a valid AWS command?

aws glue create-schema --cli-input-json '{"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName": "'$modelClassName'","DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": "'"$SCHEMA_DEFINITION"'"}' 

Gives the error:

An error occurred (InvalidInputException) when calling the CreateSchema operation: Schema definition of AVRO data format is invalid: Unexpected character ('n' (code 110)): was expecting double-quote to start field name at [Source: (String)"{name:hello}" 

Here is how I set the variable: SCHEMA_DEFINITION={"name":"Hello"}

10
  • What exact value do you need the variable to have? Should it be {name:Hello} or {"name":"Hello"}?
    – terdon
    CommentedJun 15, 2022 at 13:57
  • @terdon I would need {"name":"Hello"}, the first is not valid json for the command which is what the error seems to be telling me as far as I can tellCommentedJun 15, 2022 at 13:58
  • Is the service expecting that value to be a JSON object, or a JSON object encoded as a string ("{\"name\":\"Hello\"}")?
    – Kusalananda
    CommentedJun 15, 2022 at 14:08
  • @Kusalananda I tried setting SCHEMA_DEFINITION="{\"name\":\"Hello\"}" Then with command: aws glue create-schema --cli-input-json '{"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName": "'$modelClassName'","DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": "'$SCHEMA_DEFINITION'"}' And it tells me invalid JSON Error parsing parameter 'cli-input-json': Invalid JSON received.CommentedJun 15, 2022 at 14:14
  • @MarcoOrlando Yes, that should have been SCHEMA_DEFINITION='{\"name\":\"Hello\"}'. Note the single quotes.
    – Kusalananda
    CommentedJun 15, 2022 at 14:17

3 Answers 3

4

The JSON you seem to want to create should contain a JSON object encoded as a JSON string as the value for the SchemaDefinition key. You also seem to want to include the value of the shell variable modelClassName as the value for the SchemaName key.

You can create your JSON using jq like so:

json=$( jq -n -c \ --arg SchemaName "$modelClassName" \ --arg SchemaDefinition '{"name": "Hello"}' \ '{"RegistryId":{"RegistryName":"hxp-schema-registry"},"DataFormat":"AVRO","Compatibility":"FULL_ALL"} + $ARGS.named' ) 

or, for easier reading,

json=$( jq -n -c \ --arg SchemaName "$modelClassName" \ --arg SchemaDefinition '{"name": "Hello"}' ' { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "DataFormat": "AVRO", "Compatibility": "FULL_ALL" } + $ARGS.named' ) 

or even,

schema_json=$( jq -n -c --arg name 'hello' '$ARGS.named' ) json=$( jq -n -c \ --arg SchemaName "$modelClassName" \ --arg SchemaDefinition "$schema_json" ' { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "DataFormat": "AVRO", "Compatibility": "FULL_ALL" } + $ARGS.named' ) 

This correctly encodes both the schema definition JSON object and the model class name as JSON strings and inserts them in the appropriate place in the JSON document.

Then call your command:

aws glue create-schema --cli-input-json "$json" 
    2

    The shell is eating your quotes. Quotes are special for the shell, it needs them to protect other special characters like spaces:

    $ var=aaa bb cc bash: bb: command not found terdon@tpad ~ $ var="aaa bb cc" terdon@tpad ~ $ echo "$var" aaa bb cc 

    So, in your case, the shell thinks the quites are there to protect the value so it doesn't save them as part of the variable's value:

    $ SCHEMA_DEFINITION={"name":"Hello"} $ echo "$SCHEMA_DEFINITION" {name:Hello} 

    But don't worry, there's an easy fix! You can use single quotes around the variable definition:

    $ SCHEMA_DEFINITION='{"name":"Hello"}' $ echo "$SCHEMA_DEFINITION" {"name":"Hello"} 

    Or you can escape the quotes:

    $ SCHEMA_DEFINITION={\"name\":\"Hello\"} $ echo "$SCHEMA_DEFINITION" {"name":"Hello"} 

    Or you can use the printf builtin command:

    $ printf -v SCHEMA_DEFINITION '{"%s":"%s"}' "name" "Hello" $ echo "$SCHEMA_DEFINITION" {"name":"Hello"} 

    Or you can use the jo(1) utility:

    SCHEMA_DEFINITION=$(jo name=hello) 

    I would even suggest building the entire thing as a variable first:

    schema_definition='{"name":"Hello"}' modelClassName='"thisModel"' json_string='{"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName":' json_string="$json_string $modelClassName" json_string="$json_string"' "DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": '"$schema_definition" 

    Which results in:

    $ echo "$json_string" {"RegistryId": {"RegistryName": "hxp-schema-registry"},"SchemaName": "thisModel" "DataFormat": "AVRO","Compatibility": "FULL_ALL","SchemaDefinition": {"name":"Hello"} 

    So you can now just do:

    aws glue create-schema --cli-input-json "$json_string" 
    3
    • As far as I can see, they are actually using the variable in a double-quoted context already. I'm assuming the value should be a string, i.e. a JSON object encoded as a string.
      – Kusalananda
      CommentedJun 15, 2022 at 14:11
    • I have tried this but now I get the error Error parsing parameter 'cli-input-json': Invalid JSON received.CommentedJun 15, 2022 at 14:11
    • @MarcoOrlando see updated answer, does that help?
      – terdon
      CommentedJun 15, 2022 at 14:22
    2

    Terdon is right in that your json variable was set incorrectly but you are also passing it to the command incorrectly.

    You are passing:

    { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "SchemaName": "'$modelClassName'", "DataFormat": "AVRO", "Compatibility": "FULL_ALL", "SchemaDefinition": "'"$SCHEMA_DEFINITION"'" } 

    Which would expand to:

    { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "SchemaName": "'$modelClassName'", "DataFormat": "AVRO", "Compatibility": "FULL_ALL", "SchemaDefinition": "{"name":"Hello"}" } 

    Note specifically "{"name":"Hello"}".

    I would recommend using a heredoc for this but either way you need to remove the extra double quotes around the {}.

    read -rd json <<EOF { "RegistryId": { "RegistryName": "hxp-schema-registry" }, "SchemaName": "$modelClassName", "DataFormat": "AVRO", "Compatibility": "FULL_ALL", "SchemaDefinition": $SCHEMA_DEFINITION } EOF 

    Or using jo(1):

    json=$(jo RegistryId=$(jo RegistryName=hxp-schema-registry) \ SchemaName=$modelClassName \ DataFormat=AVRO \ Compatibility=FULL_ALL \ SchemaDefinition=$SCHEMA_DEFINITION) 

    Then you would call your command with a variable:

    aws glue create-schema --cli-input-json "$json" 

    Also I recommend https://jsonlint.com/ for validating your json inputs if you are running into issues like this.

    0

      You must log in to answer this question.

      Start asking to get answers

      Find the answer to your question by asking.

      Ask question

      Explore related questions

      See similar questions with these tags.