-------- Original Message --------
On 2012-07-01 22:31, Kristian Fiskerstrand wrote:
> On 2012-07-01 22:26, Stephan Beyer wrote:
> > Hi,
>
> Out of curiosity, what is the source of the BDB install? And is it
> configured with pthread support?
>
Hi, I've now tried to replicate the problems, unsuccessfully, but at
least I have a few observations.
I set up two identical virtual machines, both based on Gentoo, and both
with only 1 GB RAM.
gentoo4 aclocal # uname -a
Linux gentoo4 3.2.12-gentoo #1 SMP Wed Apr 25 00:24:25 CEST 2012 x86_64
Intel(R) Xeon(R) CPU E5620 @ 2.40GHz GenuineIntel GNU/Linux
They differ only in the BDB version installed;
Gentoo3 was set up with
CFLAGS="-march=core2 -O2 -pipe"
/var/tmp/portage/sys-libs/db-5.2.36/work/db-5.2.36/dist/configure
--prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
--mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
--sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64
--enable-compat185 --enable-dbm --enable-o_direct --without-uniquename
--enable-sql --enable-sql_codegen --disable-sql_compat
--with-mutex=x86/gcc-assembly --enable-cxx --enable-stl --disable-java
--disable-tcl --disable-test
And Gentoo4 was set up with
CFLAGS="-march=core2 -O2 -pipe"
/var/tmp/portage/sys-libs/db-5.1.29/work/db-5.1.29/dist/configure
--prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
--mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
--sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64
--enable-compat185 --enable-dbm --enable-o_direct --without-uniquename
--enable-sql --enable-sql_codegen --disable-sql_compat
--with-mutex=x86/gcc-assembly --enable-cxx --enable-stl --disable-java
--disable-tcl --disable-test
NB!! Note the mutex part of the configure line here; I made no changes
to this from the gentoo ebuild defaults, as I expect that the BDB
maintainers there have far more experience than me on this product's
configuration. However, from [0] it can seem that this is a less normal
specification. And so far I haven't tried other values for it to see how
that changes my results. I'll try to get some time to do this over the
course of the week.
--with-mutex=MUTEX
To force Berkeley DB to use a specific mutex implementation,
configure with --with-mutex=MUTEX, where MUTEX is the mutex
implementation you want. For example,
--with-mutex=x86/gcc-assembly will configure Berkeley DB to use
the x86 GNU gcc compiler based test-and-set assembly mutexes.
This is rarely necessary and should be done only when the
default configuration selects the wrong mutex implementation. A
list of available mutex implementations can be found in the
distribution file dist/aclocal/mutex.ac
For Gentoo 3 I used the regular build process, based on a dump from
keys2.kfwebs.net with 5,000 keys in each file. For Gentoo4 I used the
Fastbuild process, and to stress this a bit, I used 150,000 keys per
dump file (in short, I wouldn't recommend this, it is slow, but it
worked). Both dumps were read over NFS, which added a bit of time for
the process, but is also working very nicely. Resulting in;
----
gentoo4 sks # sks version
SKS version 1.1.3+
This version has a minimum compatibility requirement for recon of SKS 0.1.5
Compiled with BDB version 5.1.29
Detailed BDB environment statistics:
!Set Replication timestamp
5637 [0/1 0% rd 0/0 0% 10062/139763070297856], mpool buffer (alloc,
locked)
Further details can be seen by executing db5.1_stat -x in the KDB and
Ptree directories
gentoo4 sks # du -sh KDB PTree dump
1.1G KDB
151M PTree
4.7G dump
----
and for gentoo3:
gentoo3 sks # sks version
SKS version 1.1.3+
This version has a minimum compatibility requirement for recon of SKS 0.1.5
Compiled with BDB version 5.2.36
Detailed BDB environment statistics:
DB_FH_OPENED file-handle.flags
16416 [0/3 0% rd 0/278 0% !Own], mpool buffer (alloc)
Further details can be seen by executing db5.2_stat -x in the KDB and
Ptree directories
gentoo3 sks # du -sh KDB PTree dump
6.8G KDB
237M PTree
4.8G dump
----
So I'm unable to replicate the problem for now, but I'll see if I can
force it by manipulating the mutex configuration variable for building
BDB. It would be interesting if anyone having problem could check what
their setup is configured with.
See below for some more detailed data on each of the (fast)build
processes, including mutex use througout the process on my system.
[0] http://www.oracle.com/technetwork/products/berkeleydb/db-faq-095848.html
########################
Gentoo3 - Build
########################
#sksconf#
pagesize: 128
ptree_pagesize: 16
$ sks build dump/*.pgp -n 4 -cache 128
$ cp /root/sks-keyserver/sampleConfig/DB_CONFIG KDB/
$ sks cleandb
$ sks pbuild -cache 2 -ptree_cache 70
$ cp /root/sks-keyserver/sampleConfig/DB_CONFIG PTree/
.. some stats along the way...
gentoo3 KDB # db5.2_stat -x
3MB 72KB Mutex region size
3MB 144KB Mutex region max size
0 The number of region locks that required waiting (0%)
4 Mutex alignment
200 Mutex test-and-set spins
57459 Mutex initial count
57459 Mutex total count
0 Mutex max count
9058 Mutex free count
48401 Mutex in-use count
48491 Mutex maximum in-use count
Mutex counts
9058 Unallocated
1 env region
7 mpoolfile handle
31993 mpool buffer
17 mpool file bucket
16381 mpool hash bucket
1 mpool region
1 mutex region
gentoo3 KDB # du -sh
1.9G .
gentoo3 KDB # db5.2_stat -x
3MB 72KB Mutex region size
3MB 144KB Mutex region max size
0 The number of region locks that required waiting (0%)
4 Mutex alignment
200 Mutex test-and-set spins
57459 Mutex initial count
57459 Mutex total count
0 Mutex max count
5738 Mutex free count
51721 Mutex in-use count
51721 Mutex maximum in-use count
Mutex counts
5738 Unallocated
1 env region
7 mpoolfile handle
35313 mpool buffer
17 mpool file bucket
16381 mpool hash bucket
1 mpool region
1 mutex region
###############
Gentoo4 - Fastbuild
###############
$ sks fastbuild -n 4 -cache 128
During build
gentoo4 KDB # db5.1_stat -x
3MB 144KB Mutex region size
0 The number of region locks that required waiting (0%)
4 Mutex alignment
200 Mutex test-and-set spins
58759 Mutex total count
3008 Mutex free count
55751 Mutex in-use count
55823 Mutex maximum in-use count
Mutex counts
3008 Unallocated
1 env region
7 mpoolfile handle
39343 mpool buffer
17 mpool file bucket
16381 mpool hash bucket
1 mpool region
1 mutex region
--
----------------------------
Kristian Fiskerstrand
http://www.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Corruptissima re publica plurimæ leges
The greater the degeneration of the republic, the more of its laws
----------------------------
This email was digitally signed using the OpenPGP
standard. If you want to read more about this
The book: Sending Emails - The Safe Way: An
introduction to OpenPGP security is now
available in both Amazon Kindle and Paperback
format at
http://www.amazon.com/dp/B006RSG1S4/
----------------------------
Public PGP key 0xE3EDFAE3 at http://www.sumptuouscapital.com/pgp/
--
----------------------------
Kristian Fiskerstrand
http://www.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Corruptissima re publica plurimæ leges
The greater the degeneration of the republic, the more of its laws
----------------------------
This email was digitally signed using the OpenPGP
standard. If you want to read more about this
The book: Sending Emails - The Safe Way: An
introduction to OpenPGP security is now
available in both Amazon Kindle and Paperback
format at
http://www.amazon.com/dp/B006RSG1S4/
----------------------------
Public PGP key 0xE3EDFAE3 at http://www.sumptuouscapital.com/pgp/
signature.asc