big file and quoting

[LU Zen]

I’m trying to process a big messy csv file with the --pipe option but I keep getting errors such as unexpected EOF while looking for matching `"' or awk: cmd. line:1:  ^ unexpected newline or end of string. I suspect I’m not using the quoting correctly. 2 of the commands I’ve tried:

 

$ cat big.csv | parallel --pipe --files 'awk -v FS="\",\"" '{print $1, $3, $4, $5, $9, $14}' | grep -v "#" | sed -e '1d' -e 's/\"//g' -e 's/\/\/\//\t/g' | cut -f1-6,11 | sed -e 's/\/\//\t/g' -e 's/ /\t/g'' | parallel -Xj1 sort -k1 {} ';' rm {} > big_modified_parallel.csv

 

$ cat big.csv | parallel --pipe --files 'awk -v FS="\",\"" '{OFS="\\t\"\; print $1, $3, $4, $5, $9, $14}'  | grep -v "#" | sed -e '1d' -e 's/\"//g' -e 's/\/\/\//\t/g' | cut -f1-6,11 | sed -e 's/\/\//\t/g' -e 's/ /\t/g'' | parallel -Xj1 sort -k1 {} ';' rm {} > big_modified_parallel.csv

 

Can someone point out where I might have made a mistake?

 

Thank you.

 

 

 

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.