How to replace multiple lines with "sed" and convert to JSON?

Question

{ "auth": 'log', "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" } 24/10/2017 12:44:24 -------------------------------------------------------------------------------- { "auth": 'log', "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" } 

I need to transform a list of jsons delimited by some log lines from a file into a single json . so far i've tried this:

tac tst.txt | sed '/---------/I,+2 d' | tac > out.json

and managed to delete the log lines, but I need to replace them with a comma and add them into a list. How can I do this ? or there is any alternative for this using python ?

Answer 1

sed + jq solution:

sed -E "/^(---|[0-9][0-9])/d; s/'([^']+)'/\"\1\"/" tst.txt | jq -s '' out.json 

• /^(---|[0-9][0-9])/d - delete the unnecessary lines which start either with --- or 2 digits

• s/'([^']+)'/\"\1\"/" - replace a value in single quotes to be a valid JSON value (enclosed in double quotes)

• jq -s '' - instead of running the filter for each JSON object in the input, read the entire input stream into a large array

---

The out.json contents:

[ { "auth": "log", "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" }, { "auth": "log", "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" } ] 

Answer 2

Assuming that the quotes around log are corrected to double quotes, using sed:

sed -e '1s/^/[/' \ -e '$s/$/]/' \ -e 's/^-.*$/,/' \ -e '/^[0-9]/d' file | jq . 

The sed expressions:

1. Insert a [ in the beginning of the first line of input.
2. Insert a ] in the end of the last line of input.
3. Replace any line starting with - with a single comma.
4. Delete lines starting with any digit.

The call to jq is optional (it only sorts out the formatting).

Output:

[ { "auth": "log", "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" }, { "auth": "log", "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" } ] 

---

Short alternative using jq -s or jq --slurp:

sed -e '/^[-0-9]/d' file | jq -s