How to use shell script to split and format data output?

Question

I am trying to convert a YAML file to an HTML tables, It involves multiple complex conditions, I know this can be done using a shell script, but I have some problems in the implementation process, so come to the community for help.

The YAML content format is as follows.

- soft1: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft1_beta_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft1_alpha_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft2: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft2_beta_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft2_alpha_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip < Omit more... >

It records the historical version of multiple software, I need to convert it to HTML table code and output it to a file separately.

For example, output soft1, soft1_beta_ver, and soft1_alpha_ver to the same file (file name uses soft1), soft2 to another file.

The format of the HTML table that needs to be converted is as follows.

<table> <thead> <tr> <th>type</th> <th>ver</th> <th>link</th> </tr> </thead> <tbody> <tr> <td>soft1</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft1</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft1</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> </tbody> </table>

This is the shell script I am trying, I don't know how to split the output into multiple files, and how to get the variables of the software type.

#!/usr/bin/env bash cat << EOF <table> <thead> <tr> <th>type</th> <th>ver</th> <th>link</th> </tr> </thead> <tbody> EOF while IFS=": " read -r softver softlink do cat << EOF <tr> <td>$softver</td> <td></td> <td><a href="$softlink">download</a></td> </tr> EOF done cat << EOF </tbody> </table> EOF

Any help or suggestion on this would be greatly helpful and much appreciated.

I recommend against attempting to parse yaml files as plain text. You're not going to end up with a spec-compliant yaml interpreter too quick - you'll instead make assumptions about the data formatting , and when those assumptions are no longer true, your code will break. There are specs for this kind of thing for a reason. Your life will be much better if you use a programming language for this kind of task, python, ruby, golang are all simple suggestions, go has the advantage that it doesn't require much runtime so I'd probably go with that myself. — erik258, CommentedOct 3, 2019 at 19:56
@DanielFarrell Thanks for your reminder, it seems that the shell is not the best solution. — Matthew, CommentedOct 4, 2019 at 1:59

Ed Morton · Accepted Answer · 2019-10-03 19:29:46Z

Just generate the HTML you want from the original input file:

$ cat ../tst.awk /^-/ { sub(/:$/,"") out = type = $NF sub(/_.*/,"",out) close(out) if ( !seen[out]++ ) { prtBeg() } next } { sub(/:$/,"",$1) prtElt("<tr>") prtElt("<td>" type "</td>") prtElt("<td>" $1 "</td>") prtElt("<td>" $2 "</td>") prtElt("</tr>") } END { for (out in seen) { prtEnd() } } function prtElt(str) { depth[out] += gsub("<[^/<>]+>","&",str) printf "%*s%s\n", (depth[out]-1)*4, "", str >> out depth[out] -= gsub("</[^<>]+>","&",str) } function prtBeg() { prtElt("<table>") prtElt("<thead>") prtElt("<tr>") prtElt("<th>type</th>") prtElt("<th>ver</th>") prtElt("<th>link</th>") prtElt("</tr>") prtElt("</thead>") prtElt("<tbody>") } function prtEnd() { prtElt("</tbody>") prtElt("</table>") }

.

$ ls $ $ awk -f ../tst.awk ../file $ $ ls soft1 soft2

.

$ cat soft1 <table> <thead> <tr> <th>type</th> <th>ver</th> <th>link</th> </tr> </thead> <tbody> <tr> <td>soft1</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft1</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft1</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> </tbody> </table>

.

$ cat soft2 <table> <thead> <tr> <th>type</th> <th>ver</th> <th>link</th> </tr> </thead> <tbody> <tr> <td>soft2</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft2</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft2</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> <tr> <td>soft2_beta_ver</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft2_beta_ver</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft2_beta_ver</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> <tr> <td>soft2_alpha_ver</td> <td>V1.0.1</td> <td>http://example.com/v1.0.1.zip</td> </tr> <tr> <td>soft2_alpha_ver</td> <td>V1.0.2</td> <td>http://example.com/v1.0.2.zip</td> </tr> <tr> <td>soft2_alpha_ver</td> <td>V1.0.3</td> <td>http://example.com/v1.0.3.zip</td> </tr> </tbody> </table>

The above was run against this input file:

$ cat ../file - soft1: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft1_beta_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft1_alpha_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft2: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft2_beta_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip - soft2_alpha_ver: V1.0.1: http://example.com/v1.0.1.zip V1.0.2: http://example.com/v1.0.2.zip V1.0.3: http://example.com/v1.0.3.zip

good solution, as long as the yaml doesn't change to an equally valid but different format — erik258, CommentedOct 3, 2019 at 19:57

Thor · Accepted Answer · 2019-10-03 15:18:02Z

Something like this with sed works:

parse.sed

1r header /^-/ { s/- // s/:// h } G s/ *([^:]+): ([^\n]+)\n(.*)/ <tr>\n <td>\3<\/td>\n <td>\1<\/td>\n <td><a href="\2">Download<\/a><\/td>\n <\/tr>/p $r footer

Where header and footer contain:

header

<table> <thead> <tr> <th>type</th> <th>ver</th> <th>link</th> </tr> </thead> <tbody>

footer

 </tbody> </table>

Run it like this:

sed -Enf parse.sed infile

Output with 3 sections in infile:

<table> <thead> <tr> <th>type</th> <th>ver</th> <th>link</th> </tr> </thead> <tbody> <tr> <td>soft1</td> <td>V1.0.1</td> <td><a href="http://example.com/v1.0.1.zip">Download</a></td> </tr> <tr> <td>soft1</td> <td>V1.0.2</td> <td><a href="http://example.com/v1.0.2.zip">Download</a></td> </tr> <tr> <td>soft1</td> <td>V1.0.3</td> <td><a href="http://example.com/v1.0.3.zip">Download</a></td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.1</td> <td><a href="http://example.com/v1.0.1.zip">Download</a></td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.2</td> <td><a href="http://example.com/v1.0.2.zip">Download</a></td> </tr> <tr> <td>soft1_beta_ver</td> <td>V1.0.3</td> <td><a href="http://example.com/v1.0.3.zip">Download</a></td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.1</td> <td><a href="http://example.com/v1.0.1.zip">Download</a></td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.2</td> <td><a href="http://example.com/v1.0.2.zip">Download</a></td> </tr> <tr> <td>soft1_alpha_ver</td> <td>V1.0.3</td> <td><a href="http://example.com/v1.0.3.zip">Download</a></td> </tr> </tbody> </table>

good solution, as long as the yaml doesn't change to an equally valid but different format — erik258, CommentedOct 3, 2019 at 19:56
Thank for your answer. This script does not seem to output to multiple files. For example, output soft1, soft1_beta_ver, and soft1_alpha_ver to the same file (file name uses soft1), soft2 to another file. — Matthew, CommentedOct 4, 2019 at 2:24

Thor · Accepted Answer · 2019-10-03 15:24:13Z

You need to differentiate when the line is a heading, e.g. by "reading" 3 variables on each line:

while IFS=": " read -r a b c do if [[ "$a" == "-" ]]; then t=$b else cat << EOF <tr> <td>$t</td> <td>$a</td> <td><a href="$b:$c">download</a></td> </tr> EOF fi done

Thanks for your answer, it is very helpful!
– Matthew
CommentedOct 4, 2019 at 2:10 — Matthew, CommentedOct 4, 2019 at 2:10

Stack Exchange Network

How to use shell script to split and format data output?

3 Answers 3

You must log in to answer this question.

Linked

Hot Network Questions

How to use shell script to split and format data output?

3 Answers 3

You must log in to answer this question.

Linked

Related

Hot Network Questions