Dear developers,
The following command results in a document where valbug.txt contains multiple duplicate lines (e.g. "(TDCYXT)" without quotes)
--------
cat validation.csv | sed 's!\r\n!\n!g' | sed 's! !\n!g' | LC_ALL=C sort | uniq > valbug.txt
--------
It is definitely a problem of uniq, since the second command line of:
--------
cat validation.csv | sed 's!\r\n!\n!g' | sed 's! !\n!g' | LC_ALL=C sort > valo.txt
cat valo.txt | uniq > valbug.txt
--------
would result in the same bug and the valo.txt file is sorted correctly. In addition, the lines mentioned are equal (in hexadecimal representation) among each other in both valo.txt and valbug.txt
I also could not reduce the example since using only a few lines around the buggy lines would make it work again correctly.
You can find all files in attached zip or under https://mega.co.nz/#!PUZVgbgZ!R74BmxKcYzT_AEMwRHsMD-8CCgeRIvSnhPT-hFMihDA .
Kind regards,
Serge
Uniq Bug.zip