Re: Problem with NSAutoreleasePool in GNUstep

[ramana rao]

JP,

        First of all thanks to all GNUstep discussion group Members for
their suggestion on this, it is extremely useful for us,

        JP,  I think the problem is rightly understood.

        To elicit it again,
              Whatever memory autoreleased  within autorelease block, should
be sent sent back to the OS when the autorelease block ends.

            As others have suggested, the problem is not with Autorelease
pool, but with the program's memory management. The program is retaining the
memory up to the peak value it ever reaches.

          Some members have suggested breaking the app, so that minimum peak
memory is reached. It was difficult to implement this in our scenario. As
the memory was consumed with single library calls

    The memory consumption for archiving,  is as follows for an 100 MB
NSData
        ******************************************
        1. Conversion of NSData to pList using GMArchiver , takes up 200 MB,
for the new dictionary

         2. Conversion of the Plist to NSString , takes up 100 MB, for the
new string

        3. Conversion of NSString to UTF8 NSData , takes up 500 MB
           ( [theStr dataUsingEncoding:NSUTF8StringEncoding
allowLossyConversion:NO]; )


*************************************************************************

    As noted above the third step of converting takes up 5 times the memory.
We can't break the third statement. How ever I noticed one additive property
is holding for UTF8 conversion. I used this property to break this up.

It is like this, if we have a 10 MB  which should be converted to UTF8
NSData.

There are two ways for this

1. Directly convert the 10 MB string to UTF8 NSData
2. Break 10 MB string in to smaller peices, find UTF8 NSDAta for each peice,
join all peices together to get the total UTF8 NSData

   The resultant UTF8 NSData in both steps are identical!

With this procedure, it is taking only 100 MB as per above description

  Coming back to the originAL problem, since we don't find any way the peak
size memory of a program is sent back to the OS. So we are planning to adopt
the procedure for minimizing the peak size memory


thanks to every body!!

Ramana Rao









----- Original Message -----
From: "JP Young" <jpyoung@io.com>
To: "'Nicola Pero'" <n.pero@mi.flashnet.it>; "'ramana rao'"
<nalluri@orillion.com>
Cc: "'Richard Frith-Macdonald'" <richard@brainstorm.co.uk>; "'discuss
GNUstep'" <discuss-gnustep@gnu.org>; "'Vijaya Bhaskar Reddy K'"
<bhaskar@orillion.com>; "'Ravindra K S'" <ravindra@orillion.com>; "'Kotesh M
V'" <kotesh@orillion.com>; "'Krishna Kumar'" <krishna@orillion.com>
Sent: Monday, December 03, 2001 10:47 PM
Subject: RE: Problem with NSAutoreleasePool in GNUstep


> I not be fully informed on the original issue, but I think the source of
> this question has to do with an application (WarmStartServer) that will be
> encoding a very large list of elements to transfer to another process in
one
> large message. Let's say that the size of the element list we are encoding
> is about 50M in memory and we call NSArchiver to encode this. At our
> application level, we don't have a loop to put a pool within (though one
of
> some form certainly exists within NSArchiver to walk and encode the list).
> We are just making a single call with a root object that will cause a lot
of
> encoding to happen and generate temporary memory usage that greatly
exceeds
> the original size of what we are encoding. Putting a pool around the
> encodeRootObject: call will be closing the gate after the memory has been
> allocated :-)
>
> I think our choices to work around this problem are:
>
> 1. Try putting autorelease pools in the encodeWithCoder: methods of our
> objects that are being encoded within the list and see if this solves the
> problem. This should work if the majority of the memory allocation is
> happening in these methods.
>
> 2. Break up the list into smaller portions to encode and send. This may be
> complicated by the fact that we sometimes want to perform this same
encoding
> and write it to disk instead of send it to another process. It would be
much
> cleaner, in both cases, to just encode one big blob.
>
> 3. Go into the NSArchiver code (or write categories for it) that will use
> autorelease pools during the execution of its existing logic.
>
> I certainly agree with everyone that has said this is not a problem with
> AutoreleasePools. If there is any problem (though this is certainly not
> demonstrated out by the example code), it is with NSArchiver and its
> behavior with encoding large sets of objects.
>
> Richard's and Wim's explanation of the way that memory allocation works is
> spot-on. Once we hit that peak size, we will never grow anymore, unless we
> have to encode an even bigger object set. The peak size itself is the
issue,
> and we can't easily rearrange memory to let go of all of the huge "dead
> spots" that get created when we do this operation.
>
> Ramana, please reply, especially if I am wrong about what triggered this
> question.
>
> -- J. P.
>
> -----Original Message-----
> From: Nicola Pero [mailto:n.pero@mi.flashnet.it]
> Sent: Monday, December 03, 2001 8:14 AM
> To: ramana rao
> Cc: Richard Frith-Macdonald; discuss GNUstep; Vijaya Bhaskar Reddy K;
> Ravindra K S; Kotesh M V; Krishna Kumar; jpyoung
> Subject: Re: Problem with NSAutoreleasePool in GNUstep
>
>
>
> > Richard,
> >
> > thanks for the response.
> >
> > We tried with +freecache, still the memory used by the program is not
> going
> > down.
> >
> > I still could not get why memory usage is not coming down after [pool
> > release]. In OpenStep the memory usage does come down?
> >
> > This problem is not with particular machine, it is coming on all
machines
>
> I tried your code ... hacked it to run on my debian gnu/linux box - and I
> get -
>
> nicola@didone:~/hh/A$ ./shared_obj/ix86/linux-gnu/gnu-gnu-gnu/memory
>
> Dic 03 15:02:03 memory[2337] Memory Usage at the beginning: VmSize:
> 3928 kB
>
> Dic 03 15:02:03 memory[2337] Memory Usage before allocation: VmSize:
> 4496 kB
>
> Dic 03 15:02:05 memory[2337] Memory Usage after allocation: VmSize:
> 29604 kB
>
> Dic 03 15:02:05 memory[2337] Memory Usage after removal: VmSize:
> 29604 kB
>
> Dic 03 15:02:05 memory[2337] Memory Usage after release of the pool:
VmSize:
> 8872 kB
>
> Dic 03 15:02:05 memory[2337] Memory Usage after freeCache: VmSize:
> 4492 kB
>
> so I don't see the effect you see ... memory is returned on my system.
>
>
> Anyway - the suggestion you got from other people is correct - if your
> application reaches a peak memory usage of 1 Gb or whatever - did I hear
> correctly ?, are you sure you can't rework your application so that the
> peak memory usage is less than that ?  It's going to be healthy even not
> considering this question.  I think you really need to cut that peak down
> if you can.
>
> Typically, if you have a loop, you need to recreate the autorelease pool
> periodically, as in
>
> /* -*-objc-*- */
>
> #include <Foundation/Foundation.h>
> #include "OIProcessStatistics.h"
>
> void printMemory (NSString *when)
> {
>   NSLog (@"Memory Usage %@: %@", when,
> [OIProcessStatistics getCurrentProcessMemoryUsage]);
> }
>
>
> int main (void)
> {
>   NSAutoreleasePool * pool = [NSAutoreleasePool new];
>
>   printMemory (@"at the beginning");
>
>   {
>     NSAutoreleasePool * poolInner = [NSAutoreleasePool new];
>     NSMutableArray *theArr = [NSMutableArray new];
>     int i, j;
>
>     printMemory (@"before allocation");
>
>     for (i = 0, j =0; i < 100000; i++, j++)
>       {
> if (j = 10000)
>   {
>     DESTROY (poolInner);
>     poolInner = [NSAutoreleasePool new];
>     j = 0;
>   }
> [theArr addObject:
>   [NSString stringWithFormat:
>
@"Helllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllooooooo
> oooooooooooooooooooooooooooooo %d",
>    i]];
>       }
>
>     printMemory (@"after allocation");
>
>     [theArr removeAllObjects];
>
>     printMemory (@"after removal");
>
>     RELEASE (theArr);
>     DESTROY (poolInner);
>
>     printMemory (@"after release of the pool");
>
>     [NSAutoreleasePool freeCache];
>
>     printMemory (@"after freeCache");
>   }
>
>   DESTROY (pool);
>   return 0;
> }
>
> this example is not particularly meaningful because you are not creating
> many temporary objects inside the loop _ in a real world application you
> would probably do something inside the loop, generate a lot of temporary
> objects, and releasing the pools has a greater effect on memory
> performance.  I'm not sure how much obvious is what I've been saying :-)
> but I thought I would give a practical example code in case you felt the
> suggestion of recreating autorelease pools more often was too vague.
>
>
> _______________________________________________
> Discuss-gnustep mailing list
> Discuss-gnustep@gnu.org
> http://mail.gnu.org/mailman/listinfo/discuss-gnustep