[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: patch for printf memory use tests
From: |
Bruno Haible |
Subject: |
Re: patch for printf memory use tests |
Date: |
Sat, 15 Jan 2011 14:48:41 +0100 |
User-agent: |
KMail/1.9.9 |
Hello Bruce,
> $ cat /proc/8496/maps
> 00400000-00404000 r-xp 00000000 09:00 4228629
> .../tests/test-fprintf-posix3
> 00604000-00605000 r--p 00004000 09:00 4228629
> .../tests/test-fprintf-posix3
> 00605000-00606000 rw-p 00005000 09:00 4228629
> .../tests/test-fprintf-posix3
> 7ffff7401000-7ffff7418000 r-xp 00000000 08:06 9043976
> /lib64/libpthread-2.11.2.so
> 7ffff7418000-7ffff7618000 ---p 00017000 08:06 9043976
> /lib64/libpthread-2.11.2.so
> 7ffff7618000-7ffff7619000 r--p 00017000 08:06 9043976
> /lib64/libpthread-2.11.2.so
> 7ffff7619000-7ffff761a000 rw-p 00018000 08:06 9043976
> /lib64/libpthread-2.11.2.so
> 7ffff761a000-7ffff761e000 rw-p 00000000 00:00 0
> 7ffff761e000-7ffff7774000 r-xp 00000000 08:06 9044083
> /lib64/libc-2.11.2.so
> 7ffff7774000-7ffff7974000 ---p 00156000 08:06 9044083
> /lib64/libc-2.11.2.so
> 7ffff7974000-7ffff7978000 r--p 00156000 08:06 9044083
> /lib64/libc-2.11.2.so
> 7ffff7978000-7ffff7979000 rw-p 0015a000 08:06 9044083
> /lib64/libc-2.11.2.so
> 7ffff7979000-7ffff797e000 rw-p 00000000 00:00 0
> 7ffff797e000-7ffff79d4000 r-xp 00000000 08:06 9044230
> /lib64/libm-2.11.2.so
> 7ffff79d4000-7ffff7bd3000 ---p 00056000 08:06 9044230
> /lib64/libm-2.11.2.so
> 7ffff7bd3000-7ffff7bd4000 r--p 00055000 08:06 9044230
> /lib64/libm-2.11.2.so
> 7ffff7bd4000-7ffff7bd5000 rw-p 00056000 08:06 9044230
> /lib64/libm-2.11.2.so
> 7ffff7bd5000-7ffff7bdd000 r-xp 00000000 08:06 9044289
> /lib64/librt-2.11.2.so
> 7ffff7bdd000-7ffff7ddc000 ---p 00008000 08:06 9044289
> /lib64/librt-2.11.2.so
> 7ffff7ddc000-7ffff7ddd000 r--p 00007000 08:06 9044289
> /lib64/librt-2.11.2.so
> 7ffff7ddd000-7ffff7dde000 rw-p 00008000 08:06 9044289
> /lib64/librt-2.11.2.so
> 7ffff7dde000-7ffff7dfd000 r-xp 00000000 08:06 9044076
> /lib64/ld-2.11.2.so
> 7ffff7fc8000-7ffff7fcc000 rw-p 00000000 00:00 0
> 7ffff7ffa000-7ffff7ffb000 rw-p 00000000 00:00 0
> 7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0 [vdso]
> 7ffff7ffc000-7ffff7ffd000 r--p 0001e000 08:06 9044076
> /lib64/ld-2.11.2.so
> 7ffff7ffd000-7ffff7ffe000 rw-p 0001f000 08:06 9044076
> /lib64/ld-2.11.2.so
> 7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
> 7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Thanks a lot for the long awaited data! (I had not seen it when you posted it
in the glibc bug tracker because I'm not subscribed there.)
Let me add a first column, which indicates the length of the mapping, more
precisely the number of 4K pages:
4 00400000-00404000 r-xp 00000000 09:00 4228629
.../tests/test-fprintf-posix3 text
1 00604000-00605000 r--p 00004000 09:00 4228629
.../tests/test-fprintf-posix3 rodata
1 00605000-00606000 rw-p 00005000 09:00 4228629
.../tests/test-fprintf-posix3 data
23 7ffff7401000-7ffff7418000 r-xp 00000000 08:06 9043976
/lib64/libpthread-2.11.2.so text
512 7ffff7418000-7ffff7618000 ---p 00017000 08:06 9043976
/lib64/libpthread-2.11.2.so gap
1 7ffff7618000-7ffff7619000 r--p 00017000 08:06 9043976
/lib64/libpthread-2.11.2.so rodata
1 7ffff7619000-7ffff761a000 rw-p 00018000 08:06 9043976
/lib64/libpthread-2.11.2.so data
4 7ffff761a000-7ffff761e000 rw-p 00000000 00:00 0
bss
342 7ffff761e000-7ffff7774000 r-xp 00000000 08:06 9044083
/lib64/libc-2.11.2.so text
512 7ffff7774000-7ffff7974000 ---p 00156000 08:06 9044083
/lib64/libc-2.11.2.so gap
1 7ffff7974000-7ffff7978000 r--p 00156000 08:06 9044083
/lib64/libc-2.11.2.so rodata
1 7ffff7978000-7ffff7979000 rw-p 0015a000 08:06 9044083
/lib64/libc-2.11.2.so data
5 7ffff7979000-7ffff797e000 rw-p 00000000 00:00 0
bss
86 7ffff797e000-7ffff79d4000 r-xp 00000000 08:06 9044230
/lib64/libm-2.11.2.so text
511 7ffff79d4000-7ffff7bd3000 ---p 00056000 08:06 9044230
/lib64/libm-2.11.2.so gap
1 7ffff7bd3000-7ffff7bd4000 r--p 00055000 08:06 9044230
/lib64/libm-2.11.2.so rodata
1 7ffff7bd4000-7ffff7bd5000 rw-p 00056000 08:06 9044230
/lib64/libm-2.11.2.so data
8 7ffff7bd5000-7ffff7bdd000 r-xp 00000000 08:06 9044289
/lib64/librt-2.11.2.so text
511 7ffff7bdd000-7ffff7ddc000 ---p 00008000 08:06 9044289
/lib64/librt-2.11.2.so gap
1 7ffff7ddc000-7ffff7ddd000 r--p 00007000 08:06 9044289
/lib64/librt-2.11.2.so rodata
1 7ffff7ddd000-7ffff7dde000 rw-p 00008000 08:06 9044289
/lib64/librt-2.11.2.so data
31 7ffff7dde000-7ffff7dfd000 r-xp 00000000 08:06 9044076
/lib64/ld-2.11.2.so text
4 7ffff7fc8000-7ffff7fcc000 rw-p 00000000 00:00 0
1 7ffff7ffa000-7ffff7ffb000 rw-p 00000000 00:00 0
1 7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0 [vdso]
1 7ffff7ffc000-7ffff7ffd000 r--p 0001e000 08:06 9044076
/lib64/ld-2.11.2.so rodata
1 7ffff7ffd000-7ffff7ffe000 rw-p 0001f000 08:06 9044076
/lib64/ld-2.11.2.so data
1 7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
bss
33 7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
1 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
What you can see is that in the mapping of every shared library, there is a
gap of ca. 512 pages = 2 MB which is just reserved but not mapped to any memory.
And with 4 libraries, these make up 4 * 2 MB = 8 MB. *This* is where most of the
address space is accounted for.
> $ readelf --segments *core*
Interesting. I didn't know about this command. But it also shows these gaps
(when I apply it to a random core dump).
I don't know what this 2 MB gap is meant for. "readelf --segments
/lib64/libm.so.6"
reveals that the loadable sections have an alignment of 0x200000 - which would
make sense for 2MB pages, but the dynamic loader implements the 2 MB as a gap,
not an alignment.
Anyway, the bottom line is that the dynamic loader is free to add many unwanted
and large entries to the address space. Here it was 2 MB per shared library.
It could also have been 128 MB per shared library. In this case, calling
setrlimit (RLIMIT_AS, ..) with 20 MB or 50 MB will be just as useless as with
a 10 MB argument.
I conclude that the right thing to do is to setrlimit (RLIMIT_AS,) with a value
that is computed as
(size of address space of current process before any malloc) + 10 MB.
> I think these _are_ the right things to do. Both bump the size and
> also test for bumping against it again in the future in a way that
> lets the next hapless coder know what the issue is without having
> to go through a bunch of somersaults to figure it out.
I believe we can implement a function that returns the size of the address
space for most platforms, and on the remaining platforms (AIX, HP-UX, IRIX,
OSF/1) we haven't seen these failures.
> Also, I did look at the convolutions required to determine the
> current address space size (Viz., opening /proc/self/maps and
> parsing the text). My conclusions:
>
> 1. It is Linux specific
There's different code for different platforms.
> 3. The test is designed to catch a mis-implementation of the printf
> functionality.
More generally, we're talking about the infrastructure to catch any kind
of memory leak in a library function.
> Therefore,
> it doesn't really seem like it is worth jumping through a lot
> of hoops.
I disagree. The ability to check against memory leaks is an important feature
in the test suite, and it warrants jumping through a number of hoops.
> Also, I think I mentioned here somewhere that doing a setrlimit of
> the current size to less than what is currently in use "may" cause
> setrlimit to return EINVAL, if the implementers choose to do so.
> They did not so choose for Linux. It'd have made this much easier.
This is not a big problem: We could call setrlimit, mmap 1 page, and
munmap it if that succeeded. This gives us an upper or lower bound for
the size of the address space. It's a clever idea. Let's see how portable
it is...
I'll go on with trying the two possible implementations of a function
/* Return the current address space size of the current process. */
extern intptr_t get_ru_as (void);
a) with OS dependent code
b) with repeated setrlimit, mmap, munmap.
When this is done, we can change the memory leak tests to use this
function.
Bruno
- patch for printf memory use tests, Bruce Korb, 2011/01/14
- Re: patch for printf memory use tests, Bruno Haible, 2011/01/14
- Re: patch for printf memory use tests, Bruce Korb, 2011/01/14
- new module 'get-rusage-as', Bruno Haible, 2011/01/16
- Re: new module 'get-rusage-as', Bruno Haible, 2011/01/18
- Re: new module 'get-rusage-as', Bruce Korb, 2011/01/18
- Re: new module 'get-rusage-as', Bruno Haible, 2011/01/26
- Re: new module 'get-rusage-as', Bruno Haible, 2011/01/23
- Re: new module 'get-rusage-as', Bruno Haible, 2011/01/23