Filter columns in string with awk piped with xargs

Question

I have some files:

file1.csv file2.csv file3.csv 

A given script processes them, logging to this file:

my.log 

with this format: (filenamecol2col3):

file1.csv 1 a file2.csv 1 a file3.csv 1 a file2.csv 2 b file1.csv 2 b file3.csv 2 b file1.csv 3 c file2.csv 3 c file3.csv 3 c file2.csv 4 d file3.csv 4 d 

I'd like to get one col3 value (only last one) from the my.log file for each *.csv file.

I run this command:

ls *.csv | xargs -I@ bash -c "cat my.log | grep @ | tail -n 1 | awk '{ print $3 }'" 

It works well, except that awk is giving me all the columns.

file1.csv 3 c file2.csv 4 d file3.csv 4 d 

How could I get only col3 column? For example, this:

c d d 

Answer 1

In your expression

 "cat my.log | grep @ | tail -n 1 | awk '{ print $3 }'" 

...the double-quotes around this string mean that the single-quotes are treated as literals. They don't protect $3 from the shell, so it's being expanded as an environment variable. As $3 is not actually defined by the shell (unless this is in a script that you've invoked with 3 arguments), it becomes the empty string, and the awk expression is simply { print }, printing the whole line.

You could fix this by escaping the $:

ls *.csv | xargs -I@ bash -c "cat my.log | grep @|tail -n 1|awk '{print \$3}'" 

...or by moving the awk out of the xargs expression:

ls *.csv | xargs -I@ bash -c "cat my.log | grep @|tail -n 1"|awk '{print $3}' 

Answer 2

Piping the output of ls into xargs is a bad idea (in fact, doing anything with the output of ls other than simply viewing it in your terminal is a bad idea). If you absolutely must do something like this, at least use something like find . -maxdepth 1 -type f -iname '*.csv' -print0 and pipe that into xargs -0r.

But, in thise case, you don't need to do it at all because the filenames of your .csv files are already in my.log.

In awk:

#!/usr/bin/awk -f { seen[$1] = $3 } END { for (f in seen) { print seen[f] }; } 

or as a one-liner:

$ awk '{seen[$1] = $3}; END {for (f in seen) { print seen[f] };}' my.log c d d 

These will print the last value seen in column 3 for each file listed in column 1.

If you want it to print only the first value seen in column 3, change it to:

!seen[$1] { seen[$1] = $3 } 

---

If you don't want to use find | xargs and you really need to use the filenames of all the .csv files currently in the current directory, one alternative is to do something like this:

#!/usr/bin/perl use strict; my $logfile=shift; # get the first arg (the logfile name) my $re=join("|",@ARGV); # turn the remaining args into a regular expression @ARGV=$logfile; # set the logfile name as the sole cmd-line argument. my %seen=(); while(<>) { next unless (m/^($re)/o); # ignore any filenames that weren't on the cmd line. my(@F) = split; $seen{$F[0]} = $F[2]; # perl arrays start from 0, not 1. }; foreach my $file (sort keys %seen) { print $seen{$file}, "\n"; }; 

save it as, e.g. nandro.pl, make it executable with chmod +x and run it as:

$ ./nandro.pl my.log *.csv c d d