Re: Preview: portable dumper

[Burton Samograd]

Excellent idea and good work.  Will this also allow for the dumping of Emacs at 
arbitrary time after it’s been run, or will there still be issues with that?

Burton Samograd

> On Nov 28, 2016, at 12:50 PM, Daniel Colascione <address@hidden> wrote:
> 
> I've been working on a portable dumper for GNU Emacs. The attached patch is 
> an early version of this work. I'll write up all the usual NEWS entries, 
> changelogs, and (in this case necessary) dedicated documentation before I 
> land it. I want to start getting review comments now that the code has 
> roughly its final shape.
> 
> The point of this gargantuan patch is that we can rip out our unexec 
> implementations and replace them with loading a data file that contains an 
> Emacs heap image. There are no dependencies on executable rewriting, 
> disabling ASLR, or saving and restoring internal malloc state. This system 
> works with fully position-independent executables and with any malloc 
> implementation.
> 
> Basically, there's a new dump-emacs-portable function that walks the Emacs 
> heap and writes the data, along with necessary relocation, to a file called 
> emacs.pdmp. On startup, early in main, we find emacs.pdmp and load it. Once 
> we've loaded the file, we walk the list of relocations contained in the dump 
> and adjust it to account for the runtime locations of Emacs and the dump data 
> (neither of which we know in advance in a PIE world.)
> 
> There are a few subtleties: I've carefully designed the file format to be 
> mmap-able and to minimize the number of on-demand copies the system makes 
> while accessing this file. For example, we stick bool-vectors and string data 
> at the end of the dump in a contiguous block. We follow this block with the 
> relocations, which we can throw away as soon as we've used them.
> 
> An additional optimization follows, although this part isn't implemented yet: 
> we can define a "preferred load address" for the dump and write relocation 
> information such that if the dump and Emacs end up being where we expect them 
> to be, we don't have to perform any relocations at all.
> 
> The system gracefully degrades though. If we can't use mmap or whatever on a 
> given platform, it's possible to just slurp the whole file into a malloced 
> region of memory and access it from there. This approach would benefit from 
> compression, which will reduce IO loads: LZ4 reduces the dump size for me 
> from ~12MB to ~4MB. As in the mmap case, we can throw away
> 
> The code isn't even close to optimized yet --- I've only tested it at -O0, 
> defined GC_CHECK_MARKED_OBJECTS, and not yet inlined frequently-called 
> functions pdumper.h --- but even so, it's within 100ms or so of an unexeced 
> Emacs.
> 
> It's also possible to dump an already-dumped Emacs, so it should be possible 
> for users to have their own dump files.
> 
> If we want to preserve the current model of a single "emacs" executable that 
> contains itself, we can embed emacs.pdmp inside the emacs executable data 
> section pretty easily. There's no behavior change involved.
> 
> If you want to try this code, build CANNOT_DUMP=1, run ./temacs -l loadup 
> pdump, then ./emacs (or if that doesn't work, ./emacs --dump-file=emacs.pdmp).
> <pdumper.diff>