bug#12907: Possible Bug in sort core utility

[Coffey, Terrence (Terrence) **CTR**]

Hi Bob, Eric and Paul,
Thank you all for your quick responses. Due to time zone differences, I had 
gone home so I did not see
your emails until this morning. I'm based in Galway, Ireland. I'm 5 to 8 hours 
ahead of you depending where
in USA you are.  I started putting together a response to your email but then 
got switched to another task
and only got a chance to work on it again today. I realize that you may now be 
out of office due to Thanks
Giving holiday.

I've pulled together all your responses into this one email.
I've run most of the commands you supplied and have attached them to this email.

Unfortunately, none of your suggestions worked for me.
The result I was expecting was
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 .esd-500
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc2
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:15 .tc4
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:18 .tc3
4 drwxrwxrwt. 2 root   root   4096 Nov 19 16:05 .ICE-unix
4 -r--r--r--. 1 root   root     11 Nov 19 16:05 .X0-lock
4 drwxrwxrwt. 2 root   root   4096 Nov 19 16:05 .X11-unix
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 pulse-g9SlRO3UEqGg
4 drwx------. 2 gdm    gdm    4096 Nov 19 16:05 pulse-Jkx9hhtHj4dy
4 drwx------. 2 gdm    gdm    4096 Nov 19 16:06 orbit-gdm

See test data in attached files.


The --time-style=long.iso seem to alter the way in which the time field is 
handled in the sort command. From
my results, it appears the XX:YY (XX=Hours YY=Minutes) are treated as separate 
fields without this option.
See attached file for this.


I have a few questions
1) Should LC_ALL be set to 'C' or C or are they equivalent      export 
LC_ALL='C'
   I've tried both way and it appeared not to alter the result.

Please find attached the commands I ran with the locale unset , set to C and 
set to 'C'.

Eric, you mentioned that RH 6.3 is an old release. Where can I get the latest 
approved RH release ?
RH 6.3 is the latest release that Avaya has approved for internal use but given 
my reported issue
It would be useful to use --debug to see what is happening or not happening.

Thanks for all of your assistance,
Terence Coffey


=======
Paul Eggert address@hidden
Most likely it's a locale problem.
Try "sort --debug" and try setting LC_ALL='C'
in your environment.


========

tag 12907 notabug
thanks

On 11/16/2012 11:06 AM, Coffey, Terrence (Terrence) **CTR** wrote:
> Hi,
> I think I might have located a bug. I'm using Redhat 6.3.

Thanks for the report.  However, I suspect that you are hitting this FAQ, and 
not a bug in sort itself:
https://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021

Unfortunately, the 'sort' version shippted in RHEL 6.3 is too old to support 
the --debug option, but upgrading to a newer version will help shed some 
insight into your problems.

> address@hidden tmp]$ ls -Asl  /tmp |sort -k11 -k7M -k8n -k9,10

As I don't have the same 'ls -Asl /tmp' output, I took your output and put in 
in a temp file of my own.  Now, on that file, and with the sort
8.15 in Fedora 17, I get:

$ LC_ALL=en_US.UTF-8 sort --debug -k11 -k7M -k8n -k9,10 tmpfile
sort: using `en_US.UTF-8' sorting rules
sort: leading blanks are significant in key 1; consider also specifying `b'
sort: key 3 is numeric and spans multiple fields
sort: leading blanks are significant in key 4; consider also specifying `b'
total 76
        ^ no match for key
        ^ no match for key
        ^ no match for key
        ^ no match for key
________
0 -rw-------. 1 root   root      0 Nov  9 15:06 yum.log
                                                       ^ no match for key
                                   ___
                                        _
                                         ______________ 
_______________________________________________________
4 -r--r--r--. 1 root   root     11 Nov  9 17:37 .X0-lock
                                                        ^ no match for key
                                   ___
                                        _
                                         _______________ 
________________________________________________________

...

Wow - just that much information is quite useful.  You aren't sorting what you 
thought you were sorting.

> 
> The filename is key 11 so I was expecting to see all the dot files 
> group together

The filename is NOT field 11, but field 10 (fields start with 1, not 0).
 Furthermore, the debug option is warning you that unterminated -k 
specifications are risky; for example, it is better to use -k7,7M than -k7M.  
Also, it is warning you that -k11 includes leading space, which may affect the 
sort (depending on your locale).  Paul already pointed out that your choice of 
locale may be important (en_US.UTF-8 typically does a dictionary sort, ignoring 
punctuation, but '.' is punctuation).

> I'm try to get a listing of dot file sorted by date and all other files 
> sorted by date. I'd like all the dot file to appears before all other files.

Here's something closer, but it still has a fatal flaw - the moment you request 
to sort by dot files first, you have specified that key takes priority over 
dates.

$ LC_ALL=C sort -k10b -k7,7M -k8,8nb -k9,9 tmpfile total 76
4 drwxrwxrwt. 2 root   root   4096 Nov  9 18:35 .ICE-unix
4 -r--r--r--. 1 root   root     11 Nov  9 17:37 .X0-lock
4 drwxrwxrwt. 2 root   root   4096 Nov  9 18:35 .X11-unix
4 -r--r--r--. 1 root   root     11 Nov  9 18:35 .X2-lock
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 .esd-500
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc2
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:18 .tc3
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:15 .tc4
4 drwx------. 2 root   root   4096 Nov  9 18:35 keyring-kjLYun
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 keyring-nQ6lMb
4 drwx------. 2 WFUser WFUser 4096 Nov  9 18:20 orbit-WFUser
4 drwx------. 2 gdm    gdm    4096 Nov  9 17:39 orbit-gdm
4 drwx------. 2 root   root   4096 Nov  9 18:35 orbit-root
4 drwx------. 2 gdm    gdm    4096 Nov  9 17:40 pulse-Jkx9hhtHj4dy
4 drwx------. 2 root   root   4096 Nov  9 18:35 pulse-TSaTZvcbd1kc
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 pulse-g9SlRO3UEqGg
4 drwx------. 2 root   root   4096 Nov  9 18:35 ssh-BRSEI10523
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 virtual-WFUser.AVmQkN
0 -rw-------. 1 root   root      0 Nov  9 15:06 yum.log

Really, the ONLY way you are going to sort two independent chunks of your list 
by date is to instead concatenate two independently sorted chunks, and realize 
that ls already does the date sorting that you desire.  Perhaps you are really 
looking for:

(set -f; ls -lsd --sort=date /tmp/.[!.] /tmp/.?*; ls -ls --sort=date /tmp)

-- 
Eric Blake   address@hidden    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


-----Original Message-----
From: Bob Proulx [mailto:address@hidden 
Sent: 16 November 2012 18:59
To: Coffey, Terrence (Terrence) **CTR**
Cc: address@hidden
Subject: Re: bug#12907: Possible Bug in sort core utility

tag 12907 + moreinfo
thanks

Coffey, Terrence (Terrence) **CTR** wrote:
> I think I might have located a bug. I'm using Redhat 6.3.

What locale are you using?  You can print out your locale settings with the 
locale command.

  $ locale

Very often the locale is set to a "human" locale where case is folded and 
punctuation is ignored in "dictionary" style.  This ordering is a libc system 
library definition.

> I'm try to get a listing of dot file sorted by date and all other 
> files sorted by date. I'd like all the dot file to appears before all 
> other files.

I personally would do this with two different commands.  First list the dot 
files using shell file glob matching only dot files.  Second list non-dot 
files.  I would also use the --time-style=long-iso format so that the dates are 
printed consistently.

  $ ls -Ald /tmp/.[!.]* ; ls -Ald /tmp/*

But I realize that sets different columns for the two different commands.

> Initially I was testing on 6.2 and noticed that the .(dot) files were 
> not passed to sort. However this issue is fixed in 6.3.

I have no idea what you are talking about.  I don't recall any bug in that 
area.  And if it is there then bringing up a fixed bug muddies the water.

> However it does not appear to be able to sort dot files. The filename 
> is key 11

In your example the filename is key 10 not 11.

  0 -rw-------. 1 root   root      0 Nov  9 15:06 yum.log
  1 2           3 4      5         6 7    8 9     10

That might be the most of your problem.

> If you need some additional data, just let me know.

Try this:

  $ ls -Asl --time-style=long-iso / | env LC_ALL=C sort -k9,9 -k7,8

Then please report back in if your problem is resolved.  Please keep the bug 
log in the recipient list.

See also the FAQ entry:

  
http://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021

Bob

ls_result_no_locale.txt
Description: ls_result_no_locale.txt

LC_ALLequalsC.txt
Description: LC_ALLequalsC.txt

LC_AllequalsQuoteCQuote.txt
Description: LC_AllequalsQuoteCQuote.txt