[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Pan-devel] musings on memory consumption

From: K. Haley
Subject: Re: [Pan-devel] musings on memory consumption
Date: Thu, 29 Jul 2004 02:13:39 -0600
User-agent: Mozilla Thunderbird 0.7.2 (Windows/20040707)

Hi Anatoly,

I hope your still reading this list. I'm not one of the devs but here are my thoughts.

Anatoly Vorobey wrote:

I ended up with a patch that allows me to browse a 1-million-headers newsgroup comfortably on my machine, which is more or less what I needed. Basically, I use refcounted strings and normalised subjects. There's a new string type, RString, which stores unique strings only once by using a global hash table and a refcount field to keep track of how many times the string was referenced, allowing it to be freed when the refcount drops to 0.
Sounds usefull. I can certainly make use of it in the db code I've been working on for Pan. This way I won't have to implement something similar myself.

After this, starting Pan and loading a 1 million headers newsgroup results in about 340Mb memory used, which is tolerable for me. The slowest reactions now come from the GTK header pane, presumably because it finds it hard to cope with such a large tree. The widget spends around 10 seconds initialising or freeing the entire header pane (when entering/leaving the group). I'm not sure whether dynamic feeding of data into the widget, along the lines Evan Martin suggested in a recent message, would speed that up.
Pan currently uses GtkCTree for the article list. I don't know if it can pull in data as needed or if it requires its own copy of all string. The code will be converted to GtkTreeView as part of the rewrite but it currently has a performance problem. After adding a new GtkTreeModel it will display the visible rows then set about validating every row in the model. Even on my small group list, only 43000 rows, it makes my system un-usable for about 20 seconds. Here is a bug report on what appears to be the same issue. It also mentions the proper fix but no-one has coded it yet.

Anyway, I understand that the work I've done might not be useful to official Pan development if it's been decided to focus on moving to a DB backend. I did it primarily for myself, to scratch my own private itch. If, however, there's interest from project maintainers, and they think it might be considered for inclusion, or if other people want to try it out, do let me know; I can find some time to clean it up, remove debugging junk, test it some more and put it up for download.

By all means post it. I would suggest opening an enhancement request on bugzilla and attaching two thing. First a zip containing just your RString files for those of us attempting our own back end rewrites. Second a patch against cvs with all your RString changes, including the RString files. That way people that want to try it can. It is posible that it could make it into cvs as a temporary solution given how long the rewrite is likely to take.

Attachment: signature.asc
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]