[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Swarm-Support] RE: Huge memory allocation
|
From: |
Russell Standish |
|
Subject: |
Re: [Swarm-Support] RE: Huge memory allocation |
|
Date: |
Wed, 25 Aug 2004 13:25:44 +1000 |
|
User-agent: |
Mutt/1.5.4i |
C language code can always be compiled with icc and linked with
gcc. Issues arise with C++. However, C++ programming is not convenient
with Swarm anyway - see the paper I did with Richard Leow last year for one
way of doing this - another possibility is with SWIG, which I'm going
to be looking into.
Most swarm models will be written in Obj-C which must be compiled with
gcc. One could rewrite compute intensive parts in C, compile with icc
and link, but it is not a transparent process...
Cheers
On Tue, Aug 24, 2004 at 09:14:08PM -0400, Paul Johnson wrote:
> Russell Standish wrote:
> >
> >I assume similar considerations apply for Swarm, although (for obvious
> >reasons) Swarm cannot make use of icc.
> >
>
> I did not recall hearing of icc before, but I did some looking and I
> wondered if it is certain that it could not link programs against Swarm.
> Here's a page, am I reading it wrong? It appears to me some of you
> guys (not me, but the ones who understand Makefiles and stuff like that)
> could make these work together:
>
> http://www.intel.com/support/performancetools/c/linux/sb/cs-007709.htm
>
> Can C/C++ language object files created with the GNU gcc* compiler be
> linked into applications created with the Intel? compilers for Linux*?
>
> C language object files created with the GNU gcc compiler can be linked
> into applications created with the Intel compilers for Linux. The same C
> language calling conventions are used and the GNU glibc C language
> library is used.
>
> Starting with the Intel? C++ Compiler for Linux* 8.0, C++ language
> object files created with the gcc* 3.2 compiler can be linked into
> applications created with the Intel compiler. The files created by the
> Intel? C++ Compiler must be compiled and linked with the -cxxlib-gcc
> compiler option. This ensures that the Intel C++ Compiler uses the gcc*
> 3.2 header files and runtime libraries for compatibility.
>
> pj
>
>
> >On Tue, Aug 24, 2004 at 05:02:07PM +0100, Rouilly, Vincent wrote:
> >
> >>************************************************************************
> >>************************************************
> >>Hi,
> >>
> >>Thanks for the different answers I got concerning my problem of
> >>allocating a huge 2d space 5000x5000 + a big population of agents
> >>750000.
> >>At the moment, I am running my Swarm model under winXP, 2.66Ghz P4, 512
> >>MB RAM.
> >>
> >>I have done some tests and I managed to instantiate a world of 3000x2700
> >>2D grid + 2 Discrete2D planes + 1 million agents ! The simulation was
> >>running slow but was running without crash or noticeable memory leaks.
> >>
> >>But as soon as I wanted to have a 3000x2800 2D grid + (same parameters
> >>as before), I had a crash. Here is a copy of the Dump I got:
> >>
> >>------------------------------------------------------------------------
> >>---------------------------
> >> 198 [sig] cart 1000 handle_exceptions: Exception:
> >>STATUS_ACCESS_VIOLATION
> >>Exception: STATUS_ACCESS_VIOLATION at eip=61009ACE
> >>eax=00000000 ebx=0000065C ecx=0000065C edx=00000007 esi=04F8FE9C
> >>edi=00000000
> >>ebp=04F8FE84 esp=04F8FE6C
> >>program=c:\Swarm-2.1.1\CartSimu\myCart190804\cart.exe
> >>cs=001B ds=0023 es=0023 fs=0038 gs=0000 ss=0023
> >>Stack trace:
> >>Frame Function Args
> >>04F8FE84 61009F35 (00000000, 00000000, 00000000, 0000073C)
> >>04F8FEB4 61009F35 (00000000, 00000000, 00000000, 00000000)
> >>04F8FF04 6100B3C7 (00000006, 00000000, 00000000, 000001F4)
> >>04F8FF64 6103E747 (00000000, 00000001, 04F8FFB4, 6100324C)
> >>04F8FFB4 61003252 (61080890, 00000001, 02A40003, 61080890)
> >>990132 [sig] cart 1000 handle_exceptions: Exception:
> >>STATUS_ACCESS_VIOLATION
> >>1013736 [sig] cart 1000 handle_exceptions: Error while dumping state
> >>(probably corrupted stack)
> >>------------------------------------------------------------------------
> >>--------------------------
> >>
> >>To me, these results look quite promessing.
> >>
> >>Thanks for the comments I got, I will also have a look at EcoLab and try
> >>to have a comparative study on Swarm and Ecolab concerning the behaviour
> >>of my model, robustness and speed.
> >>
> >>Regards,
> >>
> >>Vincent.
> >>
> >>************************************************************************
> >>***********************************************
> >>
> >>
> >>Today's Topics:
> >>
> >> 1. huge memory allocation (Rouilly, Vincent)
> >> 2. Re: infinite loop (address@hidden)
> >> 3. Re: huge memory allocation (Bill Northcott)
> >> 4. Re: huge memory allocation (Ken Cline)
> >>
> >>
> >>----------------------------------------------------------------------
> >>
> >>Message: 1
> >>Date: Fri, 20 Aug 2004 18:14:32 +0100
> >>From: "Rouilly, Vincent" <address@hidden>
> >>Subject: [Swarm-Support] huge memory allocation
> >>To: <address@hidden>
> >>Message-ID: <address@hidden>
> >>Content-Type: text/plain; charset="us-ascii"
> >>
> >>Hi,
> >>
> >>First I would like to thank the people who are involved in Swarm for the
> >>work that has been done. The platform is great and it is really helpful
> >>in my research.
> >>
> >>My question:
> >>
> >>For my perfect simulation, I would like to create a 5000x5000 Grid2d
> >>space + 3 Discrete2D planes.
> >>
> >>Within this space, I would like to insert up to 750000 agents .... (huge
> >>I know, and all as to be instantiated at the same time and for the same
> >>amount of time).
> >>
> >>Of course I have memory allocation problems at the moment.
> >>
> >>How could I handle such a simulation ? Is there a way to increase the
> >>memory allocation capacity of Swarm (or my computer) ? Are they any
> >>limitations within swarm I should be aware of ? Using a cluster with
> >>Swarm is possible ?
> >>
> >>Many thanks,
> >>
> >>Vincent
> >>
> >>-------------- next part --------------
> >>An HTML attachment was scrubbed...
> >>URL:
> >>http://www.swarm.org/pipermail/support/attachments/20040820/5e15e4dd/att
> >>achment-0001.htm
> >>
> >>------------------------------
> >>
> >>------------------------------
> >>
> >>Message: 3
> >>Date: Sat, 21 Aug 2004 08:30:16 +1000
> >>From: Bill Northcott <address@hidden>
> >>Subject: Re: [Swarm-Support] huge memory allocation
> >>To: Swarm Support <address@hidden>
> >>Message-ID: <address@hidden>
> >>Content-Type: text/plain; charset="us-ascii"
> >>
> >>On 21/08/2004, at 3:14 AM, Rouilly, Vincent wrote:
> >>
> >>>My question:
> >>>
> >>>For my perfect simulation, I would like to create a 5000x5000 Grid2d
> >>>space + 3 Discrete2D planes.
> >>>
> >>>Within this space, I would like to insert up to 750000 agents ....
> >>>(huge I know, and all as to be instantiated at the same time and for
> >>>the same amount of time).
> >>>
> >>>Of course I have memory allocation problems at the moment.
> >>>
> >>>How could I handle such a simulation ? Is there a way to increase the
> >>>memory allocation capacity of Swarm (or my computer) ? Are they any
> >>>limitations within swarm I should be aware of ? Using a cluster with
> >>>Swarm is possible ?
> >>>
> >>
> >>Some more information would be helpful:
> >>What is you current system? - CPU/RAM/OS etc
> >>How big can you go? - Does 500x500x3 with 75000 agents work?
> >>
> >>That way you can see the scale of your problem. My back of the
> >>envelope feel is that there is no way you will get all that in VM on a
> >>32 bit machine. Your talking 100 simultaneous HDTV displays with
> >>every pixel calculated!
> >>
> >>Maybe it work on a 64 bit platform, but you might need truckloads of
> >>DRAM to stop the paging making everything horribly slow, and of course
> >>there is the small issue of cost. It may still be horribly slow.
> >>
> >>Otherwise, you try to box clever and see if you can break it into
> >>subproblems with looser coupling. That is also what you would need to
> >>do to run on a cluster. Swarm has no builtin cluster support, but
> >>maybe you could help add it.
> >>
> >>Finally you just might have to lower your expectations.
> >>Bill Northcott
> >>-------------- next part --------------
> >>A non-text attachment was scrubbed...
> >>Name: not available
> >>Type: text/enriched
> >>Size: 1635 bytes
> >>Desc: not available
> >>Url :
> >>http://www.swarm.org/pipermail/support/attachments/20040821/5c9be247/att
> >>achment-0001.bin
> >>
> >>------------------------------
> >>
> >>Message: 4
> >>Date: Fri, 20 Aug 2004 16:19:20 -0700 (PDT)
> >>From: Ken Cline <address@hidden>
> >>Subject: Re: [Swarm-Support] huge memory allocation
> >>To: Swarm Support <address@hidden>
> >>Message-ID: <address@hidden>
> >>Content-Type: text/plain; charset=us-ascii
> >>
> >>
> >>--- Bill Northcott <address@hidden> wrote:
> >>
> >>
> >>>On 21/08/2004, at 3:14 AM, Rouilly, Vincent wrote:
> >>>
> >>>>My question:
> >>>>
> >>>>For my perfect simulation, I would like to create a 5000x5000 Grid2d
> >>>>space + 3 Discrete2D planes.
> >>>>
> >>>>Within this space, I would like to insert up to 750000 agents ....
> >>>>(huge I know, and all as to be instantiated at the same time and for
> >>
> >>>>the same amount of time).
> >>>>
> >>>>Of course I have memory allocation problems at the moment.
> >>>>
> >>>>How could I handle such a simulation ? Is there a way to increase
> >>>>the
> >>>>memory allocation capacity of Swarm (or my computer) ? Are they any
> >>>>limitations within swarm I should be aware of ? Using a cluster with
> >>
> >>>>Swarm is possible ?
> >>>>
> >>>
> >>>Some more information would be helpful:
> >>>What is you current system? - CPU/RAM/OS etc
> >>>How big can you go? - Does 500x500x3 with 75000 agents work?
> >>>
> >>>That way you can see the scale of your problem. My back of the
> >>>envelope feel is that there is no way you will get all that in VM on a
> >>
> >>>32 bit machine. Your talking 100 simultaneous HDTV displays with
> >>>every pixel calculated!
> >>>
> >>>Maybe it work on a 64 bit platform, but you might need truckloads of
> >>>DRAM to stop the paging making everything horribly slow, and of course
> >>
> >>>there is the small issue of cost. It may still be horribly slow.
> >>>
> >>>Otherwise, you try to box clever and see if you can break it into
> >>>subproblems with looser coupling. That is also what you would need to
> >>
> >>>do to run on a cluster. Swarm has no builtin cluster support, but
> >>>maybe you could help add it.
> >>>
> >>>Finally you just might have to lower your expectations.
> >>>Bill Northcott
> >>
> >>Another thought (which you've probably already considered)...
> >>
> >>If your agents exist in only a fixed number of possible states[1] then
> >>you could possibly minimize the memory consumption by following the
> >>state machine design pattern. You'd still need at least one pointer per
> >>agent, ie the handle to its current state, and hence you'll still likely
> >>need to lower those expectations a bit. =:-(
> >>
> >>Are you convinced that qualitatively different dynamics will emerge at
> >>the 10^5 scale which don't exist with fewer agents? If not, then perhaps
> >>a semantics-based solution will work, eg rename the Agent class as
> >>"100Agents". Just a joke, but if your model has some type of agent
> >>(affiliation) hierarchy then that a natural place to begin cutting it
> >>up. (See previous discussions on this list related to clustering and
> >>parallelization of models.)
> >>
> >>Ken.
> >>________
> >>[1] Of course, any program (that terminates) has a finite number
> >> of states... the tricky part is enumerating them. =8-]
> >>
> >>
> >>
> >>=====
> >>_________________________________________________________
> >>Ken Cline W: (443) 287-2636
> >>address@hidden
> >>
> >>
> >>------------------------------
> >>
> >>_______________________________________________
> >>Support mailing list
> >>address@hidden
> >>http://www.swarm.org/mailman/listinfo/support
> >>
> >>
> >>End of Support Digest, Vol 2, Issue 8
> >>*************************************
> >>
> >>_______________________________________________
> >>Support mailing list
> >>address@hidden
> >>http://www.swarm.org/mailman/listinfo/support
> >
> >
> >
> >------------------------------------------------------------------------
> >
> >_______________________________________________
> >Support mailing list
> >address@hidden
> >http://www.swarm.org/mailman/listinfo/support
>
>
> --
> Paul E. Johnson email: address@hidden
> Dept. of Political Science http://lark.cc.ku.edu/~pauljohn
> 1541 Lilac Lane, Rm 504
> University of Kansas Office: (785) 864-9086
> Lawrence, Kansas 66044-3177 FAX: (785) 864-5700
> _______________________________________________
> Support mailing list
> address@hidden
> http://www.swarm.org/mailman/listinfo/support
--
*PS: A number of people ask me about the attachment to my email, which
is of type "application/pgp-signature". Don't worry, it is not a
virus. It is an electronic signature, that may be used to verify this
email came from me if you have PGP or GPG installed. Otherwise, you
may safely ignore this attachment.
----------------------------------------------------------------------------
A/Prof Russell Standish Director
High Performance Computing Support Unit, Phone 9385 6967, 8308 3119 (mobile)
UNSW SYDNEY 2052 Fax 9385 6965, 0425 253119 (")
Australia address@hidden
Room 2075, Red Centre http://parallel.hpc.unsw.edu.au/rks
International prefix +612, Interstate prefix 02
----------------------------------------------------------------------------
pgpbo7ndI_ois.pgp
Description: PGP signature