How to grep netcat output

Question

I'm trying to grep live text stream from netcat, but it doesn't work for me:

netcat localhost 9090 | grep sender 

returns nothing, but I'm sure that it should.

If I redirect the netcat output to a file and add some delays (simulate real environment) - then it works:

$ (sleep 5; cat netcat_output; sleep 5) | grep sender {"jsonrpc":"2.0","method":"GUI.OnScreensaverDeactivated","params":{"data": "shuttingdown":false},"sender":"xbmc"}} 

I also tried to add --line-buffered but w/o success.

What I do wrong?

Edit:

I noticed the same issue with sed, the output is empty.

But, for instance, hexdump converts text to hex live:

$ netcat localhost 9090 | hexdump -C 00000000 7b 22 6a 73 6f 6e 72 70 63 22 3a 22 32 2e 30 22 |{"jsonrpc":"2.0"| 00000010 2c 22 6d 65 74 68 6f 64 22 3a 22 50 6c 61 79 65 |,"method":"Playe| 00000020 72 2e 4f 6e 50 6c 61 79 22 2c 22 70 61 72 61 6d |r.OnPlay","param| 00000030 73 22 3a 7b 22 64 61 74 61 22 3a 7b 22 69 74 65 |s":{"data":{"ite| 00000040 6d 22 3a 7b 22 69 64 22 3a 36 2c 22 74 79 70 65 |m":{"id":6,"type| 00000050 22 3a 22 6d 6f 76 69 65 22 7d 2c 22 70 6c 61 79 |":"movie"},"play| 00000060 65 72 22 3a 7b 22 70 6c 61 79 65 72 69 64 22 3a |er":{"playerid":| 00000070 31 2c 22 73 70 65 65 64 22 3a 31 7d 7d 2c 22 73 |1,"speed":1}},"s| 

Answer 1

Answer is here: grep not matching in nc output

netcat outputs verbose logs to standard error, so we need to capture errors before we pipe to grep.

$ netcat -zv localhost 1-9000 2>&1 | grep succeeded 

Answer 2

You could use the read command (bash builtin) to force characters to be read one by one :

netcat localhost 9090 | ( cnt=0 line= while read -N 1 c; do line="$line$c" if [ "$c" = "{" ]; then cnt=$((cnt+1)) elif [ "$c" = "}" ]; then cnt=$((cnt-1)) if [ $cnt -eq 0 ]; then printf "%s\n" "$line" line= fi fi done ) | grep sender 

This script should print every full output with balancing {and }, but you can change the script to do whatever you want. This script would NOT do well on a benchmark compared to pretty much anything, but it's pretty simple and seems to work for me...

Note that your test sample didn't have matching {and }, so if this is the case of the real input, you might want to change the criteria to print the line.

Answer 3

I think the issue is the absence of newlines in the netcat output. I can see two workarounds:

1. Insert a newline every x seconds (with unfortunate consequences if the newline is inserted in the middle of source):

   ( while sleep 1; do echo; done & netcat ... ) | grep source 

2. Use awk with an RS other than newline:

   netcat ... | awk -v RS='}' -v ORS='}' '/source/' 

Answer 4

use watch command:

watch 'netcat localhost 9090 | grep sender' 

Answer 5

As per @user43791 comment, try disabling buffering in pipe, e.g.:

stdbuf -o0 netcat localhost 9090 | grep sender 

Answer 6

In Bash you can use pseudo-device files to open a TCP connection and grep it as usual. For example:

$ grep "sender" < /dev/tcp/example.com/1234 

---

To test it, simply run server which can send some file, like:

$ nc -vl 1234 < /etc/hosts 

Then in another terminal run the test to grep the output:

$ grep "127" < /dev/tcp/localhost/1234 127.0.0.1 localhost