Re: [Pan-users] 64 bits fails as solution to large binary groups

[Duncan]

Ron Johnson posted on Tue, 11 Oct 2011 09:02:38 -0500 as excerpted:

> On 10/11/2011 12:33 AM, Duncan wrote:
>> Ron Johnson posted on Mon, 10 Oct 2011 08:19:51 -0500 as excerpted:
>>
>>> On 10/10/2011 04:55 AM, Duncan wrote:

>>>> The biggest problem is pan's assumption that it has all the
>>>> information necessary to maintain its threading structure in memory
>>>> at all times.

>>> Two years ago, I suggested using SQLite.  "No!!!  It sucks over NFS!!"
>>> and "Use 64 bits" were the responses.
>>
>> Umm.. could you point out the threads?

> July 2009 for the NFS suggestion.

> Late March 2011 for "Use 64 bits".

Thanks.  I found them (and actually did a new followup on the 2009 thread 
to a post mentioning kmail, now that kmail akonadified =:^(  ).

The "It sucks over NFS" thing was indeed as you said, but keep in mind 
that it was two years ago.  AFAIK firefox now either uses its bundled 
sqlite, or if the system sqlite is used, the build tests for (and bails 
if not found) support of a couple of newer sqlite options that make it 
somewhat more robust (and secure).

Similarly, while I dumped the akonadified kmail, I know that it didn't 
make the sqlite backend the default until a number of fixes were 
introduced in upstream sqlite, making it more robust and thread-safe, 
among other things.  (And the previous akonadi default backend, mysql, 
was and remains a bit of a nightmare for a number of reasons which 
basically boil down to the fact that mysql is intended for use by 
sysadmins willing and able to take the time to understand the database 
they are working with and to optimize tweak and maintain it thru 
upgrades, etc.  While it might indeed be a great database solution under 
those circumstances, it was and is *NOT* a good solution for the "I just 
want it to work" masses, and it has proved rather troublesome indeed in 
that role.)

Based on this rather wider and more critical deployment, sqlite has 
matured substantially in the last couple years, and could arguably be a 
reasonable solution now, even if it wasn't before.

But I don't really know whether the NFS thing has been fully addressed 
with it or not, tho I believe few will argue that it hasn't gotten 
substantially better.  Still, as others argued in the thread back then, I 
don't think it's as widely deployed for the folks that pan targets as the 
one making the argument would believe.  NFS-deployed $HOME may be common 
for some, but I can't see that it's likely to be common for pan users, 
and to the extent that it is, I'd hope that the additional robustness 
gains in sqlite in the past couple years, plus the fact that the widely 
deployed firefox is already using it and has been for years, will 
mitigate the issues that were there back in 2009.

Plus, the "no due to NFS" position was hotly debated, even then, and 
didn't seem particularly well supported by other users.  So while it was 
definitely brought up as a "NO!" for that one particular user, the far 
bigger barrier was and remains as I said, we've seen a lot of talk, but 
to paraphrase a common saying, in the FLOSS world, coders (those who 
actually write the code/patches) rule, discussions drool, and so far, 
facts are facts, we've seen lots of talk and no patches.

Tho as I said there were proof of concept patches for an sqlite backend 
for the old C pan code, before the rewrite, and I still don't know why 
Charles didn't choose to go that way.  Maybe it was that he was 
uncomfortable with database coding, or that the improvements he made he 
thought were good enough, or that the database solutions at the time just 
weren't satisfactory, or a combination of all of the above, but whatever, 
it didn't happen in the rewrite and that did surprise a number of 
observers definitely including me.


The 64-bit thread, however, was more as I stated, directly, at least as I 
read it.  64-bit wasn't necessarily stated as a full solution; it was 
simply pointed out that the particular crash scenario and app memory 
usage at the crash indicated that the particular barrier you ran into at 
that point was the 32-bit address space barrier.


But regardless, as I pointed out, the last rewrite was five years ago and 
we've not seen new code implementing a disk-based pan and/or sqlite 
backend yet, and he who codes, really does rule, so without anyone 
willing to do that... for all I know the same discussion will be 
repeating another five years from now! =:^\

But you did hint that you'd look at it.  Good luck, and that's NOT being 
sarcastic, as yet again, this seems to be looming as an increasingly big 
barrier to further pan improvements!


Meanwhile, as I mentioned, I'd /really/ like to have someone take a look 
at the claws-mail code and see just how it does what it does, as it's 
FAST and MEMORY-SLIM indeed, handling headers, compared to pan.  Just 
what sort of tricks does it use and can pan possibly put them to use, 
without losing the threading, etc, that pan handles way better now?  
Because however it does it, it's managing it with base plain-text message-
per-file as pan does, but apparently doing so *FAR* more efficiently, yet 
it handles the threading, etc, that seems to be pan's bugbear in stride.

It does have some sort of index files, but then so does pan.  I really 
haven't taken a look at their format, tho I know there's a FAQ entry 
about deleting them and doing a rebuild if you move files around manually 
in the filesystem.  So they might not be plain text, but they can be 
rebuilt, and I actually did just that, following the instructions when 
doing the import from kmail, and even that went rather faster than I 
expected it to.  At this point, I'm of the opinion that even if pan 
adopted the claws-mail format and for some reason had to do a full reindex 
at every startup *AND* when updating overviews/headers, it'd probably 
STILL be faster than what pan does now!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman