Re: [Swarm-Support] RE: Huge memory allocation

[Russell Standish]

All I can suggest at present is compiling with -g, and running the
model under gdb to see exactly where the code crashes.

I would be very interested to see a head-to-head comparison of Ecolab
and Swarm. I don't think anyone has implemented precisely the same
model in both systems.

Also note that in my experience, Cygwin running under windows is not a
high performance system. Ecolab models tend to run twice as fast when
running on Linux as under Cygwin on the same hardware! And compiling
Ecolab with icc rather than gcc often gives a further 50% boost.

I assume similar considerations apply for Swarm, although (for obvious
reasons) Swarm cannot make use of icc.

On Tue, Aug 24, 2004 at 05:02:07PM +0100, Rouilly, Vincent wrote:
> ************************************************************************
> ************************************************
> Hi,
> 
> Thanks for the different answers I got concerning my problem of
> allocating a huge 2d space 5000x5000 + a big population of agents
> 750000.
> At the moment, I am running my Swarm model under winXP, 2.66Ghz P4, 512
> MB RAM.
> 
> I have done some tests and I managed to instantiate a world of 3000x2700
> 2D grid + 2 Discrete2D planes + 1 million agents ! The simulation was
> running slow but was running without crash or noticeable memory leaks.
> 
> But as soon as I wanted to have a 3000x2800 2D grid + (same parameters
> as before), I had a crash. Here is a copy of the Dump I got:
> 
> ------------------------------------------------------------------------
> ---------------------------
>     198 [sig] cart 1000 handle_exceptions: Exception:
> STATUS_ACCESS_VIOLATION
> Exception: STATUS_ACCESS_VIOLATION at eip=61009ACE
> eax=00000000 ebx=0000065C ecx=0000065C edx=00000007 esi=04F8FE9C
> edi=00000000
> ebp=04F8FE84 esp=04F8FE6C
> program=c:\Swarm-2.1.1\CartSimu\myCart190804\cart.exe
> cs=001B ds=0023 es=0023 fs=0038 gs=0000 ss=0023
> Stack trace:
> Frame     Function  Args
> 04F8FE84  61009F35  (00000000, 00000000, 00000000, 0000073C)
> 04F8FEB4  61009F35  (00000000, 00000000, 00000000, 00000000)
> 04F8FF04  6100B3C7  (00000006, 00000000, 00000000, 000001F4)
> 04F8FF64  6103E747  (00000000, 00000001, 04F8FFB4, 6100324C)
> 04F8FFB4  61003252  (61080890, 00000001, 02A40003, 61080890)
>  990132 [sig] cart 1000 handle_exceptions: Exception:
> STATUS_ACCESS_VIOLATION
> 1013736 [sig] cart 1000 handle_exceptions: Error while dumping state
> (probably corrupted stack)
> ------------------------------------------------------------------------
> --------------------------
> 
> To me, these results look quite promessing.
> 
> Thanks for the comments I got, I will also have a look at EcoLab and try
> to have a comparative study on Swarm and Ecolab concerning the behaviour
> of my model, robustness and speed.
> 
> Regards,
> 
> Vincent.
> 
> ************************************************************************
> ***********************************************
> 
> 
> Today's Topics:
> 
>    1. huge memory allocation  (Rouilly, Vincent)
>    2. Re: infinite loop (address@hidden)
>    3. Re: huge memory allocation  (Bill Northcott)
>    4. Re: huge memory allocation  (Ken Cline)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 20 Aug 2004 18:14:32 +0100
> From: "Rouilly, Vincent" <address@hidden>
> Subject: [Swarm-Support] huge memory allocation 
> To: <address@hidden>
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset="us-ascii"
> 
> Hi,
> 
> First I would like to thank the people who are involved in Swarm for the
> work that has been done. The platform is great and it is really helpful
> in my research.
> 
> My question:
> 
> For my perfect simulation, I would like to create a 5000x5000 Grid2d
> space + 3 Discrete2D planes.
> 
> Within this space, I would like to insert up to 750000 agents .... (huge
> I know, and all as to be instantiated at the same time and for the same
> amount of time). 
> 
> Of course I have memory allocation problems at the moment.
> 
> How could I handle such a simulation ? Is there a way to increase the
> memory allocation capacity of Swarm (or my computer) ? Are they any
> limitations within swarm I should be aware of ? Using a cluster with
> Swarm is possible ?
> 
> Many thanks,
> 
> Vincent
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://www.swarm.org/pipermail/support/attachments/20040820/5e15e4dd/att
> achment-0001.htm
> 
> ------------------------------
> 
> ------------------------------
> 
> Message: 3
> Date: Sat, 21 Aug 2004 08:30:16 +1000
> From: Bill Northcott <address@hidden>
> Subject: Re: [Swarm-Support] huge memory allocation 
> To: Swarm Support <address@hidden>
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset="us-ascii"
> 
> On 21/08/2004, at 3:14 AM, Rouilly, Vincent wrote:
> > My question:
> >
> > For my perfect simulation, I would like to create a 5000x5000 Grid2d
> > space + 3 Discrete2D planes.
> >
> > Within this space, I would like to insert up to 750000 agents ....
> > (huge I know, and all as to be instantiated at the same time and for 
> > the same amount of time).
> >
> >  Of course I have memory allocation problems at the moment.
> >
> > How could I handle such a simulation ? Is there a way to increase the
> > memory allocation capacity of Swarm (or my computer) ? Are they any 
> > limitations within swarm I should be aware of ? Using a cluster with 
> > Swarm is possible ?
> >
> Some more information would be helpful:
> What is you current system?    - CPU/RAM/OS etc
> How big can you go?  -  Does 500x500x3 with 75000 agents work?
> 
> That way you can see the scale of your problem.  My back of the 
> envelope feel is that there is no way you will get all that in VM on a 
> 32 bit machine.   Your talking 100 simultaneous HDTV displays with 
> every pixel calculated!
> 
> Maybe it work on a 64 bit platform, but you might need truckloads of 
> DRAM to stop the paging making everything horribly slow, and of course 
> there is the small issue of cost.  It may still be horribly slow.
> 
> Otherwise, you try to box clever and see if you can break it into 
> subproblems with looser coupling.  That is also what you would need to 
> do to run on a cluster.  Swarm has no builtin cluster support, but 
> maybe you could help add it.
> 
> Finally you just might have to lower your expectations.
> Bill Northcott
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: not available
> Type: text/enriched
> Size: 1635 bytes
> Desc: not available
> Url :
> http://www.swarm.org/pipermail/support/attachments/20040821/5c9be247/att
> achment-0001.bin
> 
> ------------------------------
> 
> Message: 4
> Date: Fri, 20 Aug 2004 16:19:20 -0700 (PDT)
> From: Ken Cline <address@hidden>
> Subject: Re: [Swarm-Support] huge memory allocation 
> To: Swarm Support <address@hidden>
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=us-ascii
> 
> 
> --- Bill Northcott <address@hidden> wrote:
> 
> > On 21/08/2004, at 3:14 AM, Rouilly, Vincent wrote:
> > > My question:
> > >
> > > For my perfect simulation, I would like to create a 5000x5000 Grid2d
> > > space + 3 Discrete2D planes.
> > >
> > > Within this space, I would like to insert up to 750000 agents ....
> > > (huge I know, and all as to be instantiated at the same time and for
> 
> > > the same amount of time).
> > >
> > >  Of course I have memory allocation problems at the moment.
> > >
> > > How could I handle such a simulation ? Is there a way to increase 
> > > the
> > > memory allocation capacity of Swarm (or my computer) ? Are they any 
> > > limitations within swarm I should be aware of ? Using a cluster with
> 
> > > Swarm is possible ?
> > >
> > Some more information would be helpful:
> > What is you current system?    - CPU/RAM/OS etc
> > How big can you go?  -  Does 500x500x3 with 75000 agents work?
> > 
> > That way you can see the scale of your problem.  My back of the
> > envelope feel is that there is no way you will get all that in VM on a
> 
> > 32 bit machine.   Your talking 100 simultaneous HDTV displays with 
> > every pixel calculated!
> > 
> > Maybe it work on a 64 bit platform, but you might need truckloads of
> > DRAM to stop the paging making everything horribly slow, and of course
> 
> > there is the small issue of cost.  It may still be horribly slow.
> > 
> > Otherwise, you try to box clever and see if you can break it into
> > subproblems with looser coupling.  That is also what you would need to
> 
> > do to run on a cluster.  Swarm has no builtin cluster support, but 
> > maybe you could help add it.
> > 
> > Finally you just might have to lower your expectations.
> > Bill Northcott
> 
> Another thought (which you've probably already considered)...
> 
> If your agents exist in only a fixed number of possible states[1] then
> you could possibly minimize the memory consumption by following the
> state machine design pattern.  You'd still need at least one pointer per
> agent, ie the handle to its current state, and hence you'll still likely
> need to lower those expectations a bit. =:-(
> 
> Are you convinced that qualitatively different dynamics will emerge at
> the 10^5 scale which don't exist with fewer agents? If not, then perhaps
> a semantics-based solution will work, eg rename the Agent class as
> "100Agents".  Just a joke, but if your model has some type of agent
> (affiliation) hierarchy then that a natural place to begin cutting it
> up.  (See previous discussions on this list related to clustering and
> parallelization of models.)
> 
> Ken.
> ________
> [1] Of course, any program (that terminates) has a finite number
>     of states... the tricky part is enumerating them. =8-]
> 
> 
> 
> =====
> _________________________________________________________
>  Ken Cline                             W: (443) 287-2636
>  address@hidden
> 
> 
> ------------------------------
> 
> _______________________________________________
> Support mailing list
> address@hidden
> http://www.swarm.org/mailman/listinfo/support
> 
> 
> End of Support Digest, Vol 2, Issue 8
> *************************************
> 
> _______________________________________________
> Support mailing list
> address@hidden
> http://www.swarm.org/mailman/listinfo/support

-- 
*PS: A number of people ask me about the attachment to my email, which
is of type "application/pgp-signature". Don't worry, it is not a
virus. It is an electronic signature, that may be used to verify this
email came from me if you have PGP or GPG installed. Otherwise, you
may safely ignore this attachment.

----------------------------------------------------------------------------
A/Prof Russell Standish                  Director
High Performance Computing Support Unit, Phone 9385 6967, 8308 3119 (mobile)
UNSW SYDNEY 2052                         Fax   9385 6965, 0425 253119 (")
Australia                                address@hidden             
Room 2075, Red Centre                    http://parallel.hpc.unsw.edu.au/rks
            International prefix  +612, Interstate prefix 02
----------------------------------------------------------------------------

pgp_pcRRDPaLx.pgp
Description: PGP signature