sed: replace pattern using a list of variables only once

Question

I am new to sed and would like to know how to replace a pattern with a different variable every time

I have a txt file that looks like this:

@K3KFV:1:1109:11598:25872 @K3KFV:1:2101:22577:15247 @K3KFV:1:1110:13477:13178 @K3KFV:1:2113:23585:6859... (etc) 

In total there are 200 different lines. In addition I have another file:

ASF356_KB822565.1:1065516-1065795 TAGGTCAAGCCCTCGGTCTATTAGTATTGGTCAGCTTAATACATTGCTGCACTTACACCT CCAACCTATCTACCTTGTTGTCTTCAAGGGACCTTACTCACTTGCGTTTTGGGATATCTT ASF356_KB822565.1:1065796-1066075 CGGATAGGGACCGAACTGTCTCACGACGTTCTGAACCCAGCTCGCGTACCGCTTTAATGG GCGAACAGCCCAACCCTTGGGACCTACTTCAGCCCCAGGATGCGATGAGCCGACATCGAG ASF356_KB822565.1:1066076-1066355 CCTTTTGCCTTTACACTCTTTGAATGGTTTCCAATCATTCTGAGGTGACCTTCGAGCGCC TCCGTTACTCTTTTGGAGGCGACCGCCCCAGTCAAACTGCCCGCCTGACATTGTCCTTCA 

which also contains 200 instances of "ASF....."

What I want is to replace the line containing "ASF..." with one from "@K3KFV:....." so in the end it will look like:

@K3KFV:1:1109:11598:25872 TAGGTCAAGCCCTCGGTCTATTAGTATTGGTCAGCTTAATACATTGCTGCACTTACACCT CCAACCTATCTACCTTGTTGTCTTCAAGGGACCTTACTCACTTGCGTTTTGGGATATCTT @K3KFV:1:2101:22577:15247 CGGATAGGGACCGAACTGTCTCACGACGTTCTGAACCCAGCTCGCGTACCGCTTTAATGG GCGAACAGCCCAACCCTTGGGACCTACTTCAGCCCCAGGATGCGATGAGCCGACATCGAG @K3KFV:1:1110:13477:13178 CCTTTTGCCTTTACACTCTTTGAATGGTTTCCAATCATTCTGAGGTGACCTTCGAGCGCC TCCGTTACTCTTTTGGAGGCGACCGCCCCAGTCAAACTGCCCGCCTGACATTGTCCTTCA 

This is the shell script I have so far:

input="K3KFVfile.txt" while IFS= read -r title do sed '/ASF/c'$title'' ASF_file done < "$input" 

But instead of giving me 200 lines of @K3KFV... I get 40000 because each ASF line gets replaced with every single one of the @K3KFV lines.

Is there a way to use sed to replace a pattern using a variable only once before moving on? Is sed the correct command to use in this case?

Answer 1

If you have the GNU implementation of sed, you could use the (uppercase) R command - one of its Commands Specific to GNU sed - to read and insert a single line of the first file each time it matches a line starting with ASF in the second. Then delete the matched line:

$ sed '/^ASF/{ R K3KFVfile.txt d }' ASF_file @K3KFV:1:1109:11598:25872 TAGGTCAAGCCCTCGGTCTATTAGTATTGGTCAGCTTAATACATTGCTGCACTTACACCT CCAACCTATCTACCTTGTTGTCTTCAAGGGACCTTACTCACTTGCGTTTTGGGATATCTT @K3KFV:1:2101:22577:15247 CGGATAGGGACCGAACTGTCTCACGACGTTCTGAACCCAGCTCGCGTACCGCTTTAATGG GCGAACAGCCCAACCCTTGGGACCTACTTCAGCCCCAGGATGCGATGAGCCGACATCGAG @K3KFV:1:1110:13477:13178 CCTTTTGCCTTTACACTCTTTGAATGGTTTCCAATCATTCTGAGGTGACCTTCGAGCGCC TCCGTTACTCTTTTGGAGGCGACCGCCCCAGTCAAACTGCCCGCCTGACATTGTCCTTCA 

You can write it as a one-liner if you prefer:

sed -e '/^ASF/{R K3KFVfile.txt' -e 'd}' ASF_file 

---

Alternatively you could consider using awk:

awk 'NR==FNR{K[FNR] = $0; next} /^ASF/{$0 = K[++n]} 1' K3KFVfile.txt ASF_file 

Answer 2

This answer is a bit of a riff on @steeldriver's

If the blank lines in the ASF_file are truly empty (no whitespace), then this awk would work

awk ' NR == FNR {x[FNR] = $0; next} {$1 = x[FNR]; print} ' K3KFVfile.txt RS='' ORS='\n\n' FS='\n' OFS='\n' ASF_file 

Before it starts reading the 2nd file, I'm changing some awk variables to control how the records and fields are determined. I'm usually not a fan of this style, but it works well here. This GNU awk version is a little tidier

gawk ' NR == FNR {x[FNR] = $0; next} ENDFILE {RS = ""; ORS = "\n\n"; FS = OFS = "\n"} {$1 = x[FNR]; print} ' K3KFVfile.txt ASF_file 

Answer 3

Using awk :

awk '/^ASF/ {getline < "@K3FVfile.txt"};1' ASF_file 

---

Same thing in Perl:

perl -pe 's/^ASF.*/<STDIN>/se' ASF_file < @K3FVfile.txt 

---

Using POSIXly sed:

sed -n '/\n/bh 1{ :k3 H;1h;n /^@K3KFV/bk3 } /^ASF/g P;/\n.*\n/D s/.*\n//;th d;:h h ' @K3FVfile.txt ASF_file 

---

Using list comprehension in Python:

python3 -c 'import sys;a,b = sys.argv[1:] with open(a) as f, open (b) as g: print(*[next(f) if l.startswith("ASF") else l for l in g],sep="",end="") ' @K3FVfile.txt ASF_file 

---

Output :

@K3KFV:1:1109:11598:25872 TAGGTCAAGCCCTCGGTCTATTAGTATTGGTCAGCTTAATACATTGCTGCACTTACACCT CCAACCTATCTACCTTGTTGTCTTCAAGGGACCTTACTCACTTGCGTTTTGGGATATCTT @K3KFV:1:2101:22577:15247 CGGATAGGGACCGAACTGTCTCACGACGTTCTGAACCCAGCTCGCGTACCGCTTTAATGG GCGAACAGCCCAACCCTTGGGACCTACTTCAGCCCCAGGATGCGATGAGCCGACATCGAG @K3KFV:1:1110:13477:13178/ CCTTTTGCCTTTACACTCTTTGAATGGTTTCCAATCATTCTGAGGTGACCTTCGAGCGCC TCCGTTACTCTTTTGGAGGCGACCGCCCCAGTCAAACTGCCCGCCTGACATTGTCCTTC 

---