Re: Dumper problems and a possible solutions

[Rich Felker]

On Wed, Jun 25, 2014 at 11:15:02PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 25 Jun 2014 15:57:30 -0400
> > From: Rich Felker <address@hidden>
> > Cc: address@hidden, address@hidden
> > 
> > > But I still don't understand how you get to 400MB.  It's not that we
> > > allocate hundreds of those 700K tables for charsets.  Do you have an
> > > explanation for this?
> > 
> > Not hundreds at a time, but if the malloc operation is just positive
> > (fake-)sbrk and the free operation is a nop, hundreds of such charset
> > load operations will quickly add up.
> 
> Free operation shouldn't be a no-op, not in malloc.

Agreed. But the question was about why my quick hack took 400MB, and
the answer is that it was using a static fake-brk with malloc=sbrk and
free=nop.

> And still, there are only a few (maybe 10) times we allocate these
> 700K tables, so 400MB sound very strange to me.

In my log, I see 768k allocations occuring roughly 94 times. Is it
possible that the temacs --batch commands I'm testing (IIRC taken from
commands that were failing in leim/Makefile, but perhaps I changed it
in some way I didn't notice?) are pulling in my .emacs file, which
might be causing more charsets to be loaded?

> > > Sorry, I don't see the difficulty.  Just make malloc/realloc/free be
> > > pointers that point to gmalloc's implementation before dumping, and to
> > > the libc implementation after it.  You may need some #define to rename
> > > malloc to some other symbol, to avoid name clashes.  Am I missing
> > > something?
> > 
> > Yeah, what happens if, after dumping, the real emacs at runtime ends
> > up calling free() on one of the pre-dump pointers?
> 
> You intercept the call and do nothing.

Right, but the free pointer can't directly point to the real (libc)
free. It has to point to the wrapper that does this range-check.

> > > > No, it's less reliable. See my other posts in the thread about what
> > > > happens if you have other libraries linked and they do nontrivial
> > > > things prior to dumping (e.g. from static ctors).
> > > 
> > > But in those other posts I thought we agreed that whatever those ctors
> > > do is irrelevant, as the dumped Emacs cannot possibly use what they
> > > allocate, and those ctors will be invoked again in the dumped Emacs.
> > 
> > Those ctors are free to inspect global data. For example one might
> > contain (this sort of idiom is necessary if you can't control the
> > relative order of ctors): if (!init) { do_something(); init=1; }. In
> > that case, the dump would save the value of init, and do_something()
> > would fail to happen at runtime.
> 
> That's the same problem as with your clock_gettime, and it must be
> fixed anyway, because any ctor run at dump time is almost certainly
> picking up data that is irrelevant to the run time.

Libc could _possibly_ work around it by virtue of having full control
over the init code. For other libraries, the issue is not fixable (see
my above example with code that has to control dependency order of
ctors), and shouldn't have to be fixed. If the library is written such
that static objects have a particular nominal initial value at the
source level, it should be able to rely on that value actually being
present at runtime. Failure to provide this guarantee is a bug in the
runtime (in this case, in the tool which produced the ELF file, i.e.
unexelf.c).

Rich